Is Agile an effective way to herd the data scientists into
the production pen or just an excuse to avoid documentation and planning? What
components in Agile do we recommend for Analytics PoCs and full-fledged
projects?
So let's discuss about it.
Every organization starts with the ambitions of business and further creates roadmap of technology, people and investment needed to unlock that business potential. To unlock the objective, we go through the phase of initial discussions, understand the requirements, technical workloads like – “I need a Linux server, database, recommendation engine, tools to handle the big data...”
What is Agile
So let's discuss about it.
Every organization starts with the ambitions of business and further creates roadmap of technology, people and investment needed to unlock that business potential. To unlock the objective, we go through the phase of initial discussions, understand the requirements, technical workloads like – “I need a Linux server, database, recommendation engine, tools to handle the big data...”
Technical requirements are quite straightforward most of the
times, but analytical activity is quite vague and there is uncertainty as we
don’t know what can be the best approach to solve the problem, the amount of
time to get the best solution.
If we develop it in traditional waterfall model approach, how
it will go:
Developing
a traditional analytics project:
Let’s say we need to build a recommendation engine for users.
Use case seems pretty easy. A traditional analytics team would go endlessly
building an engine by which will use the entire user data, run CBR(content
based recommendation) or CF(Collaborative Filtering), and after a long effort
possibly providing a powerful recommendation engine which can provide near real
time recommendation to the users. In the
entire hassle free development, there was no interaction with business people.
Challenges
in Traditional Approach:
We developed the entire engine but are not sure
about the correctness of the model. What if, we used wrong data, or wrong
variables? We don’t even know if our data exploration and insights were
correct? Oops, assume stakeholders reject it and give the feedback for existing
model, as it didn’t meet the expectations. Let’s rework now. Wouldn’t it be
awesome if we could have used Agile before?
Agile approach would have played a great role here, rapid and
iterative product development and getting rapid customer feedback cycles.
Now our problem and opportunity come at the interaction of
two trends: how we can incorporate data science and analytics, which is applied
research and needs exhaustive effort on an unpredictable timeline, into the
agile application? How can analytics applications do better than traditional
waterfall approach model? How can we craft application for unknown, evolving
data models?
What is Agile
Agile
Software development focuses on the four values(from Agile
Manifesto):
- Individuals and Interactions over process and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
Engineering
Products and Engineering data science, both are different as
data
science is less deterministic. It needs lots of creativity and though
process to
derive the best approach. Agile helps to
manage those in the
cycles,
where team explore,
learn something about the data, share the
insights with the business
team/stakeholders, align the needs and approach, take the feedback and start in the same
direction.
How Agile Analytics approach unfolds
The main
difference from traditional to Agile analytics approach is using iterative
process, sharing the learnings with stakeholders, getting rapid feedbacks and
learn with new business questions and describing datasets.
A team of Data
scientists, Business analysts and other SMEs work with the stakeholders to
discuss each question until they have:
- The clear and as narrow as possible scope
- Potential datasets and variables to be used for analysis
- Questions to be answered
It is a voyoge of discovery. The below structure known as data-value pyramid explains that.
Every
project needs an investment. And building Analytics solution is generally costlier
than developing application software. As each business silo can point to a
different domain or different data source. There is high risk in the
investment.
Agile Analytics helps to minimize the risk of pursuing the blind alleys. With the iterative approach, cyclic interaction with business team, it mitigates the risk of implementing models which turns out to be garbage.
Agile Analytics helps to minimize the risk of pursuing the blind alleys. With the iterative approach, cyclic interaction with business team, it mitigates the risk of implementing models which turns out to be garbage.
References:
- Agile Data Science : Building Data Analytics applications Book
No comments:
Post a Comment