Clouds, Lakes, and Walled Cities: Frameworks for Teaching Data Science Systems

Clouds, Lakes, and Walled Cities: Frameworks for Teaching Data Science Systems

Neal E Magee, PhD

This lecture tells the story of how systems and software development have evolved over the last two decades, and how those changes necessarily impact the work of data scientists today. Over this period, systems and services have become increasingly abstracted away from physical hardware toward ephemeral, programmable, on-demand services we call "the Cloud.” Therefore, I argue that the work of developers, engineers, and data scientists must be driven entirely by code: deploying software, provisioning and consuming storage, compute, and database systems, processing and organizing data flows, and managing workloads. When derived purely from code, solutions become repeatable, iterable, and collaborative. As a result, this programmability gives data scientists the ability to design and scale workloads in new ways: event-driven processing, horizontal and auto-scaling, asynchronous queues, and declared state patterns.

Neal is currently the Director of Solutions and DevOps Services for Research Computing at UVA. He designed and taught “DS3002 Data Science Systems” and “CS4740 Cloud Computing,” has co-authored several articles on the use of containers within biomedical and genomics research, and is the principal architect of the ACCORD platform, an NSF-funded project supporting collaborative computational research on sensitive data. Before coming to the University in 2016, he worked for over a decade in industry as a developer, cloud architect, and manager of internet operations. His work supports computational research across the University, focusing on container deployments, automation, data sharing, and federated systems. He has extensive experience with public clouds, cloud native design patterns, and architecting scalable, event-driven, and fault-tolerant systems. Neal is an AWS Certified Solution Architect and Developer, a member of the AWS Customer Council, and a GitHub Global Campus Teacher.