Is Agile an effective way to herd the data scientists into
the production pen or just an excuse to avoid documentation and planning? What
components in Agile do we recommend for Analytics PoCs and full-fledged
projects?
So let's discuss about it.
Every organization starts with the ambitions of business and further creates roadmap of technology, people and investment needed to unlock that business potential. To unlock the objective, we go through the phase of initial discussions, understand the requirements, technical workloads like – “I need a Linux server, database, recommendation engine, tools to handle the big data...”
So let's discuss about it.
Every organization starts with the ambitions of business and further creates roadmap of technology, people and investment needed to unlock that business potential. To unlock the objective, we go through the phase of initial discussions, understand the requirements, technical workloads like – “I need a Linux server, database, recommendation engine, tools to handle the big data...”
Technical requirements are quite straightforward most of the
times, but analytical activity is quite vague and there is uncertainty as we
don’t know what can be the best approach to solve the problem, the amount of
time to get the best solution.
If we develop it in traditional waterfall model approach, how
it will go:
Developing
a traditional analytics project:
Let’s say we need to build a recommendation engine for users.
Use case seems pretty easy. A traditional analytics team would go endlessly
building an engine by which will use the entire user data, run CBR(content
based recommendation) or CF(Collaborative Filtering), and after a long effort
possibly providing a powerful recommendation engine which can provide near real
time recommendation to the users. In the
entire hassle free development, there was no interaction with business people.
Challenges
in Traditional Approach:
We developed the entire engine but are not sure
about the correctness of the model. What if, we used wrong data, or wrong
variables? We don’t even know if our data exploration and insights were
correct? Oops, assume stakeholders reject it and give the feedback for existing
model, as it didn’t meet the expectations. Let’s rework now. Wouldn’t it be
awesome if we could have used Agile before?
Agile approach would have played a great role here, rapid and
iterative product development and getting rapid customer feedback cycles.
Now our problem and opportunity come at the interaction of
two trends: how we can incorporate data science and analytics, which is applied
research and needs exhaustive effort on an unpredictable timeline, into the
agile application? How can analytics applications do better than traditional
waterfall approach model? How can we craft application for unknown, evolving
data models?
Agile
Software development focuses on the four values(from Agile
Manifesto):
- Individuals and Interactions over process and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
Engineering
Products and Engineering data science, both are different as
data
science is less deterministic. It needs lots of creativity and though
process to
derive the best approach. Agile helps to
manage those in the
cycles,
where team explore,
learn something about the data, share the
insights with the business
team/stakeholders, align the needs and approach, take the feedback and start in the same
direction.
How Agile Analytics approach unfolds
The main
difference from traditional to Agile analytics approach is using iterative
process, sharing the learnings with stakeholders, getting rapid feedbacks and
learn with new business questions and describing datasets.
A team of Data
scientists, Business analysts and other SMEs work with the stakeholders to
discuss each question until they have:
- The clear and as narrow as possible scope
- Potential datasets and variables to be used for analysis
- Questions to be answered
It is a voyoge of discovery. The below structure known as data-value pyramid explains that.
Every
project needs an investment. And building Analytics solution is generally costlier
than developing application software. As each business silo can point to a
different domain or different data source. There is high risk in the
investment.
Agile Analytics helps to minimize the risk of pursuing the blind alleys. With the iterative approach, cyclic interaction with business team, it mitigates the risk of implementing models which turns out to be garbage.
Agile Analytics helps to minimize the risk of pursuing the blind alleys. With the iterative approach, cyclic interaction with business team, it mitigates the risk of implementing models which turns out to be garbage.
References:
- Agile Data Science : Building Data Analytics applications Book
Nice information. Thank you for sharing such post...!
ReplyDeleteVery nice post. Awesome article... Really helpful...!
ReplyDeleteNice Blog
ReplyDeleteIot Training in Bangalore
Iteanz
usefull and awesome article.
ReplyDeleteVery Nice Blog in Data science thank you for sharing the knowledge with us:
ReplyDeletekeep blogging more thank you fort sharing
Devops Training in bangalore
very nice blog thanks for sharing
ReplyDeletevery nice blog It was useful
ReplyDeleteI got nice blog
ReplyDeletesap partner companies in bangalore
sap implementation companies in bangalore
sap partners in india
aws staffing
jquery interview questions
sql interview questions
Nice blog
ReplyDeleteuipath training in bangalore
angular4 interview questions
python interview questions
artificial intelligence interview questions
python online training
artificial intelligence online training
Excellent blog
ReplyDeletepython interview questions
git interview questions
django interview questions
sap grc interview questions and answers
advanced excel training in bangalore
zend framework interview questions
apache kafka interview questions
uipath training in bangalore
very nice interview questions
ReplyDeletevlsi interview questions
extjs interview questions
laravel interview questions
sap bi/bw interview questions
pcb interview questions
unix shell scripting interview questions
really awesome blog
ReplyDeletehr interview questions
hibernate interview questions
selenium interview questions
c interview questions c++ interview questions
linux interview questions
useful blog
ReplyDeletespring mvc interview questions
machine learning online training
servlet interview questions
mytectra.in thanks for sharing
Thank you for sharing this type of interview questions
ReplyDeleteIot Online Training
Itil Interview Questions
Salesforce Interview Questions
Msbi Interview questions
Salesforce Interview Questions
C Interview Questions
aws partners in bangalore
aws partners in india
aws service providers in india
aws service providers
sap partner companies in bangalore
ReplyDeletesap implementation companies in bangalore
sap partners in india
aws staffing
Nice Blog
ReplyDeletepython training in bangalore
python online training