(5) The Waterfall

• Traditional software engineering worked well in a static world
  • Long initial phases of data and process analyses
  • Followed by long phases of data modeling and functionality specification
  • Followed by long phases of database and functionality implementation
  • Followed by the realization that the world had changed in the meantime
• Working towards the perfect stable IT is not a reasonable goal
  • Companies are being restructured and reoriented faster than ever
  • IT as something done in isolation in one company is a thing of the past
• Design for change and see the IT infrastructure as an ecosystem
  • In order to be agile, software components must be decoupled
  • Granularity continues to shrink, for example in case of microservices

(6) Beyond the Waterfall

• Waterfall models are causing trouble on a variety of fronts
  1. All logic works directly on the one true model of the database
  2. Making changes to the overall data and functionality model is complex
  3. The overall model/implementation assumes a homogenous environment
  4. Dependencies between software modules make it hard to isolate changes
• Newer approaches try to avoid these problems with new methods and technologies
  1. Move from direct database access to a multi-tier architecture [https://en.wikipedia.org/wiki/Multitier_architecture]
  2. Use Domain-Driven Design [https://en.wikipedia.org/wiki/Domain-driven_design] for establishing and separating bounded contexts
  3. Services/APIs hide implementation technologies/details from consumers
• Loosely coupled [http://dret.net/netdret/publications#pau09a] Web Technologies are replacing traditional EAI [https://en.wikipedia.org/wiki/Enterprise_application_integration] approaches

(7) Data Lakes are EAI-ish

• Data Lakes are better than Data Warehouses
  • Retaining as much of the original model/structure as possible
  • Schemaless and thus allowing everything to be poured into it
  • Better at enabling exploration than the more goal-driven warehouses
• Data Lakes are the EAI of Data Science
  • They assume that there is one data source for all applications
  • Changes in the data source thus potentially affect many applications
  • Very centralized and heavyweight approach
• Expect decentralized architectures to replace/augment Big Data Lakes

(9) Data Lake Today

(10) Data Lake Problems

• Volume: It may be impractical/impossible to capture all data
  • IoT scenarios have high volume/rates of raw sensor data
• Velocity: Data may only be needed for local and immediate decision-making
  • Connected cars provide some data services but most data remains on-board for operations
• Variety: Integration scenarios require the harmonization of data models
  • Decision-making needs integrated data instead of vendor-specific models
• Variability: Data cleansing often is needed to improve data quality
  • Raw data can be low quality and should remain isolated from good data
• Regulation: Certain data sources or aspects may be regulated
  • Depending on regulation, the physical location of data may be important
• Experimentation: Data sources and models are constantly evolving
  • How to make sure that the lake and lake users are not tightly coupled

(11) Data Lake Future

(12) Focus on the Fish!

• Data Lakes are yet another tool to identify and harvest value
  • You can only find what's in the lake
• The lake is only a tool; what you want is just the fish!
  • Data Lakes now are a thing because for the first time, they are cost-effective
• Centralization is a common pattern for simplification
  • Big centralized services are successful and thriving
  • Internally, they often are less centralized than they look from outside
• There is no such thing as just data without a defining context
  • Every Data Lake has a purpose and is shaped by the motivation behind it
  • Keep things open and flexible but don't expect it to work for everything

(14) Schemaless and Open

• Traditional databases/warehouses are focused around models/schemas
• Data Lakes tend to be schemaless to allow greater flexibility
• There still is some underlying metamodel implicit in storage and processing
• Being schemaless and open is a big plus for adding data easily
• But there is more to data science than just having a lot of data

(15) Cascading Lakes

• Data cleansing [https://en.wikipedia.org/wiki/Data_cleansing] often is a useful first step to isolate bad data
  • Keeping the bad data may still be useful or required by law/regulation
• Regulation/legislation may require certain data to be hidden
  • There even may be requirements about physical data locations
• Isolating data by stability and permanence can help
  • Different properties of data sources can require different management regimes
• Creating, changing, and combining value chains should be easy
  • Allow for experimentation and for easily adding new sources/consumers

(16) Open Lake Landscape

• Different lakes support different kinds of fishery
  • Structured Lakes support traditional structured queries
  • Semi-structured Lakes support more document-oriented query types
  • Logical Lakes support inference and allow logic-based queries
  • Statistical Lakes are good for teaching machines how to fish
• Be wary of anybody selling you the best lake for any kind of fishery
  • Data Scientists should focus on solution architecture and patterns
  • Having a plan for how to create, connect, and re-connect lakes is important

(17) Open and Flexible

• There are many ways to catch big fish
• Plan for an evolving landscape that can change and adapt
• Data always is highly decentralized, so Big Data should be, too
• The trick is to keep architecture open, not individual solutions

(19) From Data to Services

• Data Lakes are better than Data Warehouses
• Individual Big Data Lakes are just one possibility
• Data Scientists should know about lake ecosystems
• Expect APIs to become the plumbing connecting the lakes
• Data Scientists should be fish farmers, not just fishermen

(20) Thanks! Q&A

• Me: http://dret.net/netdret/ [http://dret.net/netdret/]
• Slides online: dret.net/lectures/haas2016/ [http://dret.net/lectures/haas2016/]
• Slide sources: dret/lectures [http://github.com/dret/lectures/tree/master/haas2016] on GitHub [http://github.com/]