Browsing by Author "Bisset, Keith R."
Now showing 1 - 20 of 28
Results Per Page
Sort Options
- An Algorithm for Influence Maximization and Target Set Selection for the Deterministic Linear Threshold ModelSwaminathan, Anand (Virginia Tech, 2014-07-03)The problem of influence maximization has been studied extensively with applications that include viral marketing, recommendations, and feed ranking. The optimization problem, first formulated by Kempe, Kleinberg and Tardos, is known to be NP-hard. Thus, several heuristics have been proposed to solve this problem. This thesis studies the problem of influence maximization under the deterministic linear threshold model and presents a novel heuristic for finding influential nodes in a graph with the goal of maximizing contagion spread that emanates from these influential nodes. Inputs to our algorithm include edge weights and vertex thresholds. The threshold difference greedy algorithm presented in this thesis takes into account both the edge weights as well as vertex thresholds in computing influence of a node. The threshold difference greedy algorithm is evaluated on 14 real-world networks. Results demonstrate that the new algorithm performs consistently better than the seven other heuristics that we evaluated in terms of final spread size. The threshold difference greedy algorithm has tuneable parameters which can make the algorithm run faster. As a part of the approach, the algorithm also computes the infected nodes in the graph. This eliminates the need for running simulations to determine the spread size from the influential nodes. We also study the target set selection problem with our algorithm. In this problem, the final spread size is specified and a seed (or influential) set is computed that will generate the required spread size.
- Analysis system using brokers that access information sources(United States Patent and Trademark Office, 2018-01-16)Systems, methods, and computer-readable media for generating a data set are provided. One method includes generating a data set based on input data using a plurality of brokers. The method further includes receiving a request from a user and determining whether the request can be fulfilled using data currently in the data set. When the request can be fulfilled using data currently in the data set, the data is accessed using broker(s) configured to provide access to data within the data set. When the request cannot be fulfilled using data currently in the data set, at least one new broker is spawned using existing broker(s) and additional data needed to fulfill the request is added to the data set using the new broker. The method further includes generating a response to the request using one or more of the plurality of brokers.
- Complex situation analysis system that generates a social contact network, uses edge brokers and service brokers, and dynamically adds brokers(United States Patent and Trademark Office, 2013-04-16)A system for generating a representation of a situation is disclosed. The system comprises one or more computer-readable media including computer-executable instructions that are executable by one or more processors to implement a method of generating a representation of a situation. The method comprises receiving input data regarding a target population. The method further comprises constructing a synthetic data set including a synthetic population based on the input data. The synthetic population includes a plurality of synthetic entities. Each synthetic entity has a one-to-one correspondence with an entity in the target population. Each synthetic entity is assigned one or more attributes based on information included in the input data. The method further comprises receiving activity data for a plurality of entities in the target population.
- Complex situation analysis system that spawns/creates new brokers using existing brokers as needed to respond to requests for data(United States Patent and Trademark Office, 2014-03-25)Systems, methods, and computer-readable media for generating a data set are provided. One method includes generating a data set based on input data using a plurality of brokers. The method further includes receiving a request from a user and determining whether the request can be fulfilled using data currently in the data set. When the request can be fulfilled using data currently in the data set, the data is accessed using broker(s) configured to provide access to data within the data set. When the request cannot be fulfilled using data currently in the data set, at least one new broker is spawned using existing broker(s) and additional data needed to fulfill the request is added to the data set using the new broker. The method further includes generating a response to the request using one or more of the plurality of brokers.
- Complex situation analysis system using a plurality of brokers that control access to information sources(United States Patent and Trademark Office, 2016-06-14)Systems, methods, and computer-readable media for generating a data set are provided. One method includes generating a data set based on input data using a plurality of brokers. The method further includes receiving a request from a user and determining whether the request can be fulfilled using data currently in the data set. When the request can be fulfilled using data currently in the data set, the data is accessed using broker(s) configured to provide access to data within the data set. When the request cannot be fulfilled using data currently in the data set, at least one new broker is spawned using existing broker(s) and additional data needed to fulfill the request is added to the data set using the new broker. The method further includes generating a response to the request using one or more of the plurality of brokers.
- Computational Cost Analysis of Large-Scale Agent-Based Epidemic SimulationsKamal, Tariq (Virginia Tech, 2016-09-21)Agent-based epidemic simulation (ABES) is a powerful and realistic approach for studying the impacts of disease dynamics and complex interventions on the spread of an infection in the population. Among many ABES systems, EpiSimdemics comes closest to the popular agent-based epidemic simulation systems developed by Eubank, Longini, Ferguson, and Parker. EpiSimdemics is a general framework that can model many reaction-diffusion processes besides the Susceptible-Exposed-Infectious-Recovered (SEIR) models. This model allows the study of complex systems as they interact, thus enabling researchers to model and observe the socio-technical trends and forces. Pandemic planning at the world level requires simulation of over 6 billion agents, where each agent has a unique set of demographics, daily activities, and behaviors. Moreover, the stochastic nature of epidemic models, the uncertainty in the initial conditions, and the variability of reactions require the computation of several replicates of a simulation for a meaningful study. Given the hard timelines to respond, running many replicates (15-25) of several configurations (10-100) (of these compute-heavy simulations) can only be possible on high-performance clusters (HPC). These agent-based epidemic simulations are irregular and show poor execution performance on high-performance clusters due to the evolutionary nature of their workload, large irregular communication and load imbalance. For increased utilization of HPC clusters, the simulation needs to be scalable. Many challenges arise when improving the performance of agent-based epidemic simulations on high-performance clusters. Firstly, large-scale graph-structured computation is central to the processing of these simulations, where the star-motif quality nodes (natural graphs) create large computational imbalances and communication hotspots. Secondly, the computation is performed by classes of tasks that are separated by global synchronization. The non-overlapping computations cause idle times, which introduce the load balancing and cost estimation challenges. Thirdly, the computation is overlapped with communication, which is difficult to measure using simple methods, thus making the cost estimation very challenging. Finally, the simulations are iterative and the workload (computation and communication) may change through iterations, as a result introducing load imbalances. This dissertation focuses on developing a cost estimation model and load balancing schemes to increase the runtime efficiency of agent-based epidemic simulations on high-performance clusters. While developing the cost model and load balancing schemes, we perform the static and dynamic load analysis of such simulations. We also statically quantified the computational and communication workloads in EpiSimdemics. We designed, developed and evaluated a cost model for estimating the execution cost of large-scale parallel agent-based epidemic simulations (and more generally for all constrained producer-consumer parallel algorithms). This cost model uses computational imbalances and communication latencies, and enables the cost estimation of those applications where the computation is performed by classes of tasks, separated by synchronization. It enables the performance analysis of parallel applications by computing its execution times on a number of partitions. Our evaluations show that the model is helpful in performance prediction, resource allocation and evaluation of load balancing schemes. As part of load balancing algorithms, we adopted the Metis library for partitioning bipartite graphs. We have also developed lower-overhead custom schemes called Colocation and MetColoc. We performed an evaluation of Metis, Colocation, and MetColoc. Our analysis showed that the MetColoc schemes gives a performance similar to Metis, but with half the partitioning overhead (runtime and memory). On the other hand, the Colocation scheme achieves a similar performance to Metis on a larger number of partitions, but at extremely lower partitioning overhead. Moreover, the memory requirements of Colocation scheme does not increase as we create more partitions. We have also performed the dynamic load analysis of agent-based epidemic simulations. For this, we studied the individual and joint effects of three disease parameter (transmissiblity, infection period and incubation period). We quantified the effects using an analytical equation with separate constants for SIS, SIR and SI disease models. The metric that we have developed in this work is useful for cost estimation of constrained producer-consumer algorithms, however, it has some limitations. The applicability of the metric is application, machine and data-specific. In the future, we plan to extend the metric to increase its applicability to a larger set of machine architectures, applications, and datasets.
- Data Integration Methodologies and Services for Evaluation and Forecasting of EpidemicsDeodhar, Suruchi (Virginia Tech, 2016-05-31)Most epidemiological systems described in the literature are built for evaluation and analysis of specific diseases, such as Influenza-like-illness. The modeling environments that support these systems are implemented for specific diseases and epidemiological models. Hence they are not reusable or extendable. This thesis focuses on the design and development of an integrated analytical environment with flexible data integration methodologies and multi-level web services for evaluation and forecasting of various epidemics in different regions of the world. The environment supports analysis of epidemics based on any combination of disease, surveillance sources, epidemiological models, geographic regions and demographic factors. The environment also supports evaluation and forecasting of epidemics when various policy-level and behavioral interventions are applied, that may inhibit the spread of an epidemic. First, we describe data integration methodologies and schema design, for flexible experiment design, storage and query retrieval mechanisms related to large scale epidemic data. We describe novel techniques for data transformation, optimization, pre-computation and automation that enable flexibility, extendibility and efficiency required in different categories of query processing. Second, we describe the design and engineering of adaptable middleware platforms based on service-oriented paradigms for interactive workflow, communication, and decoupled integration. This supports large-scale multi-user applications with provision for online analysis of interventions as well as analytical processing of forecast computations. Using a service-oriented architecture, we have provided a platform-as-a-service representation for evaluation and forecasting of epidemics. We demonstrate the applicability of our integrated environment through development of the applications, DISIMS and EpiCaster. DISIMS is an interactive web-based system for evaluating the effects of dynamic intervention strategies on epidemic propagation. EpiCaster is a situation assessment and forecasting tool for projecting the state of evolving epidemics such as flu and Ebola in different regions of the world. We discuss how our platform uses existing technologies to solve a novel problem in epidemiology, and provides a unique solution on which different applications can be built for analyzing epidemic containment strategies.
- A Database Supported Modeling Environment for Pandemic Planning and Course of Action AnalysisMa, Yifei (Virginia Tech, 2013-06-24)Pandemics can significantly impact public health and society, for instance, the 2009 H1N1
and the 2003 SARS. In addition to analyzing the historic epidemic data, computational simulation of epidemic propagation processes and disease control strategies can help us understand the spatio-temporal dynamics of epidemics in the laboratory. Consequently, the public can be better prepared and the government can control future epidemic outbreaks more effectively. Recently, epidemic propagation simulation systems, which use high performance computing technology, have been proposed and developed to understand disease propagation processes. However, run-time infection situation assessment and intervention adjustment, two important steps in modeling disease propagation, are not well supported in these simulation systems. In addition, these simulation systems are computationally efficient in their simulations, but most of them have
limited capabilities in terms of modeling interventions in realistic scenarios.
In this dissertation, we focus on building a modeling and simulation environment for epidemic propagation and propagation control strategy. The objective of this work is to
design such a modeling environment that both supports the previously missing functions,
meanwhile, performs well in terms of the expected features such as modeling fidelity,
computational efficiency, modeling capability, etc. Our proposed methodologies to build
such a modeling environment are: 1) decoupled and co-evolving models for disease propagation, situation assessment, and propagation control strategy, and 2) assessing situations and simulating control strategies using relational databases. Our motivation for exploring these methodologies is as follows: 1) a decoupled and co-evolving model allows us to design modules for each function separately and makes this complex modeling system design simpler, and 2) simulating propagation control strategies using relational databases improves the modeling capability and human productivity of using this modeling environment. To evaluate our proposed methodologies, we have designed and built a loosely coupled and database supported epidemic modeling and simulation environment. With detailed experimental results and realistic case studies, we demonstrate that our modeling environment provides the missing functions and greatly enhances many expected features, such as modeling capability, without significantly sacrificing computational efficiency and scalability. - Detail in network models of epidemiology: are we there yet?Eubank, Stephen; Barrett, Christopher L.; Beckman, Richard J.; Bisset, Keith R.; Durbeck, L.; Kuhlman, Christopher J.; Lewis, Bryan L.; Marathe, Achla; Marathe, Madhav V.; Stretz, P. (Taylor & Francis, 2010)Network models of infectious disease epidemiology can potentially provide insight into how to tailor control strategies for specific regions, but only if the network adequately reflects the structure of the region’s contact network. Typically, the network is produced by models that incorporate details about human interactions. Each detail added renders the models more complicated and more difficult to calibrate, but also more faithful to the actual contact network structure. We propose a statistical test to determine when sufficient detail has been added to the models and demonstrate its application to the models used to create a synthetic population and contact network for the USA.
- Development of Person-Person Network and Interacting PTTS in EpiSimdemicsMishra, Gaurav (Virginia Tech, 2014-05-23)Communications over social media, telephone, email, text etc have emerged as an integral part of modern society and they are popularly used for the expression of anger, anxiety, fear, agitation and opinion by the people. People's social interaction tend to increase dramatically during periods of epidemics, protest and calamities. Therefore, above mentioned communication channels plays an important role in the spread of infectious phenomenon, like rumors, fads and effects. These infectious phenomena alters people's behavior during disease epidemic [1][2]. Social contact networks and epidemics co-evolve [1][2]. The spread of a disease influences people's behavior which in turn changes their social contact network, thereby altering the disease spread itself. As a result, there is a need for modeling the spread of these infectious phenomena that lead to changes in behavior. Their propagation among population primarily depends on the social contact network. The nature of social contagion spread is very similar to the spread of any infectious disease as they are contagious in nature. To spread contagious disease requires direct exposure to an infectious agent, whereas social contagions can be spread using various communications media like social networking forums, phones, emails and tweets. EpiSimdemics is an individual-based modeling environment. It uses a people-location bipartite graph as the underlying network [3]. In its current form, EpiSimdemics requires two people to interact at a location to model simulations. Thus, it cannot simulate the spread of social contagions that do not necessarily require the meeting of two agents at a location. We enhance EpiSimdemics by incorporating Person-Person network, which can model communications between people that are not contact based such as communications over email, phone, text and tweet. This Person-Person network is used to model effects (social contagion) which induce behavioral changes in population and thus impacting the disease spread. The disease spread is modeled on Person-Location network. This leads to the scenario of two interacting networks: Person-Person network modeling social contagion and Person-Location modeling disease. Theoretically, there can be multiple such networks modeling various interacting phenomena. We demonstrate the usefulness of this network by modeling and simulating two interacting PTTSs (probabilistic timed transition systems). To model disease epidemics, we have defined Disease Model and to model effects (social contagion), we have defined Fear Model. We show how these models influence each other by performing simulations on EpiSimdemics with interacting Disease and Fear Model. Therefore a model that does not include the affect adaptations on disease epidemics and vice-versa, fails to reflect the actual behavior of a society during disease epidemic spread. The addition of Person-Person network to EpiSimdemics will allow for a better understanding of the affect adaptions, which can include behavior changes in society during an epidemic outbreak. This would lead to effective interventions and help to better understand the dynamics of disease epidemic.
- A dynamic middleware to integrate multiple cloud infrastructures with remote apllicationsBhattacharjee, Tirtha Pratim (Virginia Tech, 2014-12-04)In an era with compelling need for greater computation power, the aggregation of software system components is becoming more challenging and diverse. The new-generation scientific applications are growing hub of complex and intense computation performed on huge data set with exponential growth. With the development of parallel algorithms, design of multi-user web applications and frequent changes in software architecture, there is a bigger challenge lying in front of the research institutes and organizations. Network science is an interesting field posing extreme computation demands to sustain complex large-scale networks. Several static or dynamic network analysis have to be performed through algorithms implementing complex graph theories, statistical mechanics, data mining and visualization. Similarly, high performance computation infrastructures are imbibing multiple characters and expanding in an unprecedented way. In this age, it's mandatory for all software solutions to migrate to scalable platforms and integrate cloud enabled data center clusters for higher computation needs. So, with aggressive adoption of cloud infrastructures and resource-intensive web-applications, there is a pressing need for a dynamic middleware to bridge the gap and effectively coordinate the integrated system. Such a heterogeneous environment encourages the devising of a transparent, portable and flexible solution stack. In this project, we propose adoption of Virtual Machine aware Portable Batch System Cluster (VM-aware PBS Cluster), a self-initiating and self-regulating cluster of Virtual Machines (VM) capable of operating and scaling on any cloud infrastructure. This is an unique but simple solution for large-scale softwares to migrate to cloud infrastructures retaining the most of the application stack intact. In this project, we have also designed and implemented Cloud Integrator Framework, a dynamic implementation of cloud aware middleware for the proposed VM-aware PBS cluster. This framework regulates job distribution in an aggregate of VMs and optimizes resource consumption through on-demand VM initialization and termination. The model was integrated into CINET system, a network science application. This model has enabled CINET to mediate large-scale network analysis and simulation tasks across varied cloud platforms such as OpenStack and Amazon EC2 for its computation requirements.
- Economic and Social Impact of Influenza Mitigation Strategies by Demographic ClassBarrett, Christopher L.; Bisset, Keith R.; Leidig, Jonathan; Marathe, Achla; Marathe, Madhav V. (Elsevier, 2011-03-01)Background—We aim to determine the economic and social impact of typical interventions proposed by the public health officials and preventive behavioral changes adopted by the private citizens in the event of a “flu-like” epidemic. Method—We apply an individual-based simulation model to the New River Valley area of Virginia for addressing this critical problem. The economic costs include not only the loss in productivity due to sickness but also the indirect cost incurred through disease avoidance and caring for dependents. Results—The results show that the most important factor responsible for preventing income loss is the modification of individual behavior; it drops the total income loss by 62% compared to the base case. The next most important factor is the closure of schools which reduces the total income loss by another 40%. Conclusions—The preventive behavior of the private citizens is the most important factor in controlling the epidemic.
- Estimating Reachability Set Sizes in Dynamic GraphsAji, Sudarshan Mandayam (Virginia Tech, 2014-07-01)Graphs are a commonly used abstraction for diverse kinds of interactions, e.g., on Twitter and Facebook. Different kinds of topological properties of such graphs are computed for gaining insights into their structure. Computing properties of large real networks is computationally very challenging. Further, most real world networks are dynamic, i.e., they change over time. Therefore there is a need for efficient dynamic algorithms that offer good space-time trade-offs. In this thesis we study the problem of computing the reachability set size of a vertex, which is a fundamental problem, with applications in databases and social networks. We develop the first Giraph based algorithms for different dynamic versions of these problems, which scale to graphs with millions of edges.
- A High Performance C++ Generic Benchmark for Computational EpidemiologyPugaonkar, Aniket Narayan (Virginia Tech, 2015-01-31)An effective tool used by planners and policy makers in public health, such as Center for Disease Control (CDC), to curtail spread of infectious diseases over a given population is contagion diffusion simulations. These simulations model the relevant characteristics of the population (age, gender, income etc.) and the disease (attack rate, etc.) and compute the spread under various configuration and plausible intervention strategies (such as vaccinations, school closure, etc.). Hence, the model and the computation form a complex agent based system and are highly compute and resource intensive. In this work, we design a benchmark consisting of several kernels which capture the essential compute, communication, and data access patterns for such applications. For each kernel, the benchmark provides different evaluation strategies. The goal is to (a) derive alternative implementations for computing the contagion by combining different implementation of the kernels, and (b) evaluate which combination of implementation, runtime, and hardware is most effective in running large scale contagion diffusion simulations. Our proposed benchmark is designed using C++ generic programming primitives and lifting sequential strategies for parallel computations. Together, these lead to a succinct description of the benchmark and significant code reuse when deriving strategies for new hardware. For the benchmark to be effective, this aspect is crucial, because the potential combination of hardware and runtime are growing rapidly thereby making infeasible to write optimized strategy for the complete contagion diffusion from ground up for each compute system.
- High-performance biocomputing for simulating the spread of contagion over large contact networksBisset, Keith R.; Aji, Ashwin M.; Marathe, Madhav V.; Feng, Wu-chun (BMC, 2012-04-12)Background Many important biological problems can be modeled as contagion diffusion processes over interaction networks. This article shows how the EpiSimdemics interaction-based simulation system can be applied to the general contagion diffusion problem. Two specific problems, computational epidemiology and human immune system modeling, are given as examples. We then show how the graphics processing unit (GPU) within each compute node of a cluster can effectively be used to speed-up the execution of these types of problems. Results We show that a single GPU can accelerate the EpiSimdemics computation kernel by a factor of 6 and the entire application by a factor of 3.3, compared to the execution time on a single core. When 8 CPU cores and 2 GPU devices are utilized, the speed-up of the computational kernel increases to 9.5. When combined with effective techniques for inter-node communication, excellent scalability can be achieved without significant loss of accuracy in the results. Conclusions We show that interaction-based simulation systems can be used to model disparate and highly relevant problems in biology. We also show that offloading some of the work to GPUs in distributed interaction-based simulations can be an effective way to achieve increased intra-node efficiency.
- Inferring Network Status from Partial ObservationsRangudu, Venkata Pavan Kumar (Virginia Tech, 2017-02-09)In many network applications, such as the Internet and infrastructure networks, nodes fail or get congested dynamically, but tracking this information about all the nodes in a network where some dynamical processes are taking place is a fundamental problem. In this work, we study the problem of inferring the complete set of failed nodes, when only a sample of the node failures are known---we will be referring to this particular problem as prob{} . We consider the setting in which there exists correlations between node failures in networks, which has been studied in the case of many infrastructure networks. We formalize the prob{} problem using the Minimum Description Length (MDL) principle and we show that, in general, finding solutions that minimize the MDL cost is hard, and develop efficient algorithms with rigorous performance guarantees for finding near-optimal MDL cost solutions. We evaluate our methods on both synthetic and real world datasets, which includes the one from WAZE. WAZE is a crowd-sourced road navigation tool, that collects and presents the traffic incident reports. We found that the proposed greedy algorithm for this problem is able to recover $80%$, on average, of the failed nodes in a network for a given partial sample of input failures, which are sampled from the true set of failures at some predefined rate. Furthermore, we have also proved that this algorithm will find a solution that has MDL cost with an additive approximation guarantee of log(n) from the optimal.
- A Middleware for Large-scale Simulation Systems & Resource ManagementMakkapati, Hemanth (Virginia Tech, 2013-05-26)Socially coupled systems are comprised of inter-dependent social, organizational, economic, infrastructure and physical networks. Today's urban regions serve as an excellent example of such systems. People and institutions confront the implications of the increasing scale of information becoming available due to a combination of advances in pervasive computing, data acquisition systems as well as high performance computing. Integrated modeling and decision making environments are necessary to support planning, analysis and counter factual experiments to study these complex systems. Here, we describe SIMFRASTRUCTURE -- a computational infrastructure that supports high performance computing oriented decision and analytics environments to study socially coupled systems. Simfrastructure provides a middleware with multiplexing mechanism by which modeling environments with simple and intuitive user-interfaces can be plugged in as front-end systems, and high-end computing resources -- such as clusters, grids and clouds -- can be plugged in as back-end systems for execution. This makes several key aspects of simulation systems such as the computational complexity, data management and resource management and allocation completely transparent to the users. The decoupling of user interfaces, data repository and computational resources from simulation execution allows users to run simulations and access the results asynchronously and enables them to add new datasets and simulation models dynamically. Simfrastructure enables implementation of a simple yet powerful modeling environment with built-in analytics-as-aservice platform, which provides seamless access to high end computational resources, through an intuitive interface for studying socially coupled systems. We illustrate the applicability of Simfrastructure in the context of an integrated modeling environment to study public health epidemiology and network science.
- MPI-ACC: Accelerator-Aware MPI for Scientific ApplicationsAji, Ashwin M.; Panwar, Lokendra S.; Ji, Feng; Murthy, Karthik; Chabbi, Milind; Balaji, Pavan; Bisset, Keith R.; Dinan, James; Feng, Wu-chun; Mellor-Crummey, John; Ma, Xiaosong; Thakur, Rajeev (2016-05-01)
- My4Sight: A Human Computation Platform for Improving Flu PredictionsAkupatni, Vivek Bharath (Virginia Tech, 2015-09-17)While many human computation (human-in-the-loop) systems exist in the field of Artificial Intelligence (AI) to solve problems that can't be solved by computers alone, comparatively fewer platforms exist for collecting human knowledge, and evaluation of various techniques for harnessing human insights in improving forecasting models for infectious diseases, such as Influenza and Ebola. In this thesis, we present the design and implementation of My4Sight, a human computation system developed to harness human insights and intelligence to improve forecasting models. This web-accessible system simplifies the collection of human insights through the careful design of the following two tasks: (i) asking users to rank system-generated forecasts in order of likelihood; and (ii) allowing users to improve upon an existing system-generated prediction. The structured output collected from querying human computers can then be used in building better forecasting models. My4Sight is designed to be a complete end-to- end analytical platform, and provides access to data collection features and statistical tools that are applied to the collected data. The results are communicated to the user, wherever applicable, in the form of visualizations for easier data comprehension. With My4Sight, this thesis makes a valuable contribution to the field of epidemiology by providing the necessary data and infrastructure platform to improve forecasts in real time by harnessing the wisdom of the crowd.
- Optimizing Data Accesses for Scaling Data-intensive Scientific ApplicationsYeom, Jae-seung (Virginia Tech, 2014-05-30)Data-intensive scientific applications often process an enormous amount of data. The scalability of such applications depends critically on how to manage the locality of data. Our study explores two common types of applications that are vastly different in terms of memory access pattern and workload variation. One includes those with multi-stride accesses in regular nested parallel loops. The other is for processing large-scale irregular social network graphs. In the former case, the memory location or the data item accessed in a loop is predictable and the load on processing a unit work (an array element) is relatively uniform with no significant variation. On the other hand, in the latter case, the data access per unit work (a vertex) is highly irregular in terms of the number of accesses and the locations being accessed. This property is further tied to the load and presents significant challenges in the scalability of the application performance. Designing platforms to support extreme performance scaling requires understanding of how application specific information can be used to control the locality and improve the performance. Such insights are necessary to determine which control and which abstraction to provide for interfacing an underlying system and an application as well as for designing a new system. Our goal is to expose common requirements of data-intensive scientific applications for scalability. For the former type of applications, those with regular accesses and uniform workload, we contribute new methods to improve the temporal locality of software-managed local memories, and optimize the critical path of scheduling data transfers for multi-dimensional arrays in nested loops. In particular, we provide a runtime framework allowing transparent optimization by source-to-source compilers or automatic fine tuning by programmers. Finally, we demonstrate the effectiveness of the approach by comparing against a state-of-the-art language-based framework. For the latter type, those with irregular accesses and non-uniform workload, we analyze how the heavy-tailed property of input graphs limits the scalability of the application. Then, we introduce an application-specific workload model as well as a decomposition method that allows us to optimize locality with the custom load balancing constraints of the application. Finally, we demonstrate unprecedented strong scaling of a contagion simulation on two state-of-the-art high performance computing platforms.