“Observability” has become a key trend in Service Reliability Engineering practice. One of the recommendations from Gartner’s latest Market Guide for IT Infrastructure Monitoring Tools released in January 2020 says, “Contextualize data that ITIM tools collect from highly modular IT architectures by using AIOps to manage other sources, such as observability metrics from cloud-native monitoring tools.”
Like so many other terms in software engineering, ‘observability’ is a term borrowed from an older physical discipline: in this case, control systems engineering. Let me use the definition of observability from control theory in Wikipedia: “observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.”
Observability is gaining attention in the software world because of its effectiveness at enabling engineers to deliver excellent customer experiences with software despite the complexity of the modern digital enterprise.
When we blew up the monolith into many services, we lost the ability to step through our code with a debugger: it now hops the network. Monitoring tools are still coming to grips with this seismic shift.
How is observability different than monitoring?
Monitoring requires you to know what you care about before you know you care about it. Observability allows you to understand your entire system and how it fits together, and then use that information to discover what specifically you should care about when it’s most important.
Monitoring requires you to already know what normal is. Observability allows discovery of different types of ‘normal’ by looking at how the system behaves, over time, in different circumstances.
Monitoring asks the same questions over and over again. Is the CPU usage under 80%? Is memory usage under 75% percent? Or, is the latency under 500ms? This is valuable information, but monitoring is useful for known problems.
Observability, on the other side, is about asking different questions almost all the time. You discover new things.
Observability allows the discovery of different types of ‘normal’ by looking at behavior, over time, in different circumstances.
Metrics do not equal observability.
What Questions Can Observability Answer?
Below are sample questions that can be addressed by an effective observability solution:
Why is x broken?
What services does my service depend on — and what services are dependent on my service?
Why has performance degraded over the past quarter?
What changed? Why?
What logs should we look at right now?
What is system performance like for our most important customers?”
What SLO should we set?
Are we out of SLO?
What did my service look like at time point x?
What was the relationship between my service and x at time point y?
What was the relationship of attributed across the system before we deployed? What’s it like now?
What is most likely contributing to latency right now? What is most likely not?
Are these performance optimizations on the critical path?
About the Author –
Sri is a Serial Entrepreneur with over 30 years’ experience delivering creative, client-centric, value-driven solutions for bootstrapped and venture-backed startups.
Dashboards
are considered to be one of the key success factors of businesses. Easy access
to real-time KPIs allows them to be proactive and address business challenges
before they impact the bottom line.
SAP Lumira
Designer, formerly known as SAP Design Studio, is one of the robust products
available in the market for developing top-of-the-line analytical applications
and business dashboards. However, every tool by its core functionality has
limitations for some use cases and business processes. One of such considerable
cases is the ability to write back to a database directly from the dashboard.
Write back
functionality assists business users to modify the data while analyzing from
the dashboard rather than doing it in the source system. This functionality
facilitates business users to manipulate the data and reflects refreshed data
in the dashboard for further review and assessment.
This article
introduces the use of Lumira SDK Extension component, POST RESPONSE PARSER,
which enables the core range of Lumira designer to expand its boundary to
include write-back functionality by integrating external Web API into the
Lumira Dashboard.
Integrating
Post Response Parser SDK Extension, the Lumira dashboard could be transformed
from a pure data visualization application into an interactive data management
analytical application.
Data Exploration & Smart Visualizations
Dashboards
are analytical tools that visually track, analyzes and display Key Performance
Indicators (KPIs) to the business processes or the portfolios. It provides a
comprehensive snapshot of the performance of a key component within the
portfolio. KPIs are business metrics which assists the leadership team to
arrive at key decisions and drive towards the goals.
Business
Dashboards and analytical applications provide at-a-glance visual and graphical
representation of data which eliminates the need to go through long and complex
excel spreadsheets.
Also, it’s
time-consuming and difficult to pull out the most important business
information whereas presenting that information in an appealing, visual way is
more result-driven and effective.
Interactive
dashboards enable us to visualize the data, filter on demand and simply click
to dive deeper, quickly engage end-users, and provide an intuitive experience
and insights.
Among
various visualization tools available in the market, SAP Lumira has an edge
being an SAP tool where end-user consumption of analytical applications is
governed and secured by the SAP Business Objects BI Platform.
Extending the Dashboard Functionality
Lumira
designer provides extensive customizations through scripting, styling with CSS
and above all, the integration of external SDK Components makes it a pinnacle
tool to achieve the desired functionalities.
Like any
other technology, dashboards are constantly evolving, with versatility and
impactful ability of integrating SDK components assisting the rapidly
developing scope and scale of visualizations for the organizations.
Along those
lines, Business users expect the ability to modify the data that lies behind a
visualization component by providing data inputs to the dashboard while analyzing
the data and anticipate the changes to be reflected immediately in the
dashboard.
Lumira
designer leverages support for updating or modifying the data in underlying
database through write back functionality.
Benefits of
write back in the dashboard:
It
transforms a traditional dashboard to Interactive analytical application which
supports business data modifications
It allows
data analysis and data update from the same dashboard, rather switching over
different applications for each task
SDK
Extensions
SDK stands for Software Development Kit. SDK is set of
tools, libraries, code samples, processes and guides that allows developers to
create applications on a specific platform.
SDK Extension components can be integrated flawlessly
into the core application to utilize its features for the customized product
developments. The visualization of extension components is based on HTML,
JavaScript and CSS.
Web
Application Programming Interface
Web API is an Application Programming Interface over the
web which can be accessed using HTTP protocol.
Web API is an extensible framework for building HTTP
based services that can be accessed in different applications on different
platforms such as web, windows, mobile etc.
Integrating Web APIs into the Lumira designer enhances
the dashboard functionality by adding abilities not offered in the baseline
version of the tool, such as providing the possibility of writing back to the
source database directly from the dashboard itself.
Post Response
Parser
Post Response Parser is an SDK Extension, with which you
can model your application to make AJAX (Asynchronous JavaScript and XML) calls
to any Web API and evaluate the response for desired interactivity in Lumira
designer.
Feature of Post Response Parser:
Opens a request via AJAX call to any specific URL
Accepts parameters along the Request
Supports BIAL (BI Action Language) Scripting for
interactive control at runtime
Business Use
Case
In Banking, Credit Control & Monitoring department
uses exception reports on their day to day operations for the analysis of their
customers credit performance. Based on the outcomes, the team decides on the
action to be taken for the respective customers with the various levels of
audits.
Business Team faces challenges to maintain and track the
remarks and comments on each customer by looking at the reports. So CCM wants
to develop a dashboard with the ability to update their observations and
comments on the same dashboard which in turn gets stored in database.
Lumira designer provides sub-optimal workarounds for
capturing the filters and remarks with technical components like Bookmarks and
Comments which comes along with the core application, but these components
cannot not write back to the database, but incorporating SDK Extensions along with the core would be
able to achieve the desired customization in the dashboard application.
Functionality
and Process Flow
The Post Response parser integrates external Web API into
the Lumira designer, this SDK extension passes the parameters from the
dashboard to the underlying stored procedure in Web API which in turn updates
to the database.
Snippet of process to be followed:
Install Post
Response Parser SDK Extension at client and server system
Encapsulate
the parameters as global variable and enable its property to expose as URL
Parameter
Create a Web
service for dashboard to accommodate the database updates
Define an
event to trigger the SDK Extension in Lumira application
Reload the
data source through script to reflect the changes in dashboard
Conclusion
Lumira designer is competent to build Business
Intelligence Applications that can be dynamic and customizable as per the
business users’ workflow.
An interactive prototype is the best way for both users
and designers to learn about their specific needs.
In conclusion, Lumira Designer with SDK Extensions offers that capabilities to accommodate our design process and it stands strong in its ability to build simple or complex Analytic Applications and Executive Dashboards.
About the Author:
Kashif is a SAP Business objects consultant and a business analytics enthusiast. He believes “Ultimate goal is not about winning, but to reach within the depth of capabilities and to compete against yourself to be better than what you are today.”
“Uber, the
world’s largest taxi company, owns no vehicles. Facebook, the world’s most
popular media owner, creates no content. Alibaba, the most valuable retailer,
has no inventory. Netflix, the world’s largest movie house, own no cinemas. And
Airbnb, the world’s largest accommodation provider, owns no real estate.
Something interesting is happening.”
– Tom Goodwin, an executive at the French media group Havas.
This new breed
of companies is the fastest growing in history because they own the customer
interface layer. It is the platform where all the value and profit is. “Platform
business” is a more wholesome term for this model for which data is the fuel;
Big Data & AI/ML technologies are the harbinger of new waves of
productivity growth and innovation.
With Big data and AI/ML is making a big difference in the area of public health, let’s see how it is helping us tackle the global emergency of coronavirus formally known as COVID-19.
DISCOVERING /
DETECTING
Chinese technology giant Alibaba has
developed an AI system for detecting the COVID-19 in CT scans of patients’ chests with
96% accuracy against viral pneumonia cases. It only takes 20 seconds for the AI
to decide, whereas humans generally take about 15 minutes to diagnose the
illness as there can be upwards of 300 images to evaluate. The system was trained on images and data
from 5,000 confirmed coronavirus cases and has been tested in hospitals
throughout China. Per a report, at least 100 healthcare facilities are
currently employing Alibaba’s AI to detect COVID-19.
Ping An Insurance (Group) Company of China, Ltd (Ping An) aims to address the issue of lack of radiologists by introducing the COVID-19 smart image-reading system. This image-reading system can read the huge volumes of CT scans in epidemic areas.
Ping An Smart Healthcare uses clinical
data to train the AI model of the COVID-19 smart image-reading system. The AI
analysis engine conducts a comparative analysis of multiple CT scan images of
the same patient and measures the changes in lesions. It helps in tracking the
development of the disease, evaluation of the treatment and in prognosis of
patients. Ultimately it assists doctors to diagnose, triage and evaluate
COVID-19 patients swiftly and effectively.
Ping An Smart Healthcare’s COVID-19 smart
image-reading system also supports AI image-reading remotely by medical
professionals outside the epidemic areas. Since its launch, the smart
image-reading system has provided services to more than 1,500 medical
institutions. More than 5,000 patients have received smart image-reading
services for free.
The more solutions the better. At least
when it comes to helping overwhelmed doctors provide better diagnoses and,
thus, better outcomes.
MONITORING
AI based Temperature monitoring & scanning
In Beijing, China, subway passengers are being screened for
symptoms of coronavirus, but not by health authorities. Instead, artificial
intelligence is in-charge.
Two Chinese AI giants, Megvii and Baidu, have introduced
temperature-scanning. They have implemented scanners to detect body temperature
and send alerts to company workers if a person’s body temperature is high
enough to constitute a fever.
Megvii’s AI system detects body temperatures for up to 15 people per second and up to 16 feet. It monitors as many as 16 checkpoints in a single station. The system integrates body detection, face detection, and dual sensing via infrared cameras and visible light. The system can accurately detect and flag high body temperature even when people are wearing masks, hats, or covering their faces with other items. Megvii’s system also sends alerts to an on-site staff member.
Baidu, one of the largest search-engine companies in China,
screens subway passengers at the Qinghe station with infrared scanners. It also
uses a facial-recognition system, taking photographs of passengers’ faces. If
the Baidu system detects a body temperature of at least 99-degrees Fahrenheit,
it sends an alert to the staff member for another screening. The technology can
scan the temperatures of more than 200 people per minute.
AI based Social Media Monitoring
An international team is using machine
learning to scour through social media posts, news reports, data from official
public health channels, and information supplied by doctors for warning signs of
the virus across geographies. The program is looking for social media posts
that mention specific symptoms, like respiratory problems and fever, from a
geographic area where doctors have reported potential cases. Natural language
processing is used to parse the text posted on social media, for example, to
distinguish between someone discussing the news and someone complaining about
how they feel.
The approach has proven capable of spotting
a coronavirus needle in a haystack of big data. This technique could help
experts learn how the virus behaves. It may be possible to determine the age,
gender, and location of those most at risk quicker than using official medical
sources.
PREDICTING
Data from hospitals, airports, and other
public locations are being used to predict disease spread and risk. Hospitals
can also use the data to plan for the impact of an outbreak on their
operations.
Kalman Filter
Kalman filter was pioneered by Rudolf Emil
Kalman in 1960, originally designed and developed to solve the navigation
problem in the Apollo Project. Since then, it has been applied to numerous
cases such as guidance, navigation, and control of vehicles, computer vision’s
object tracking, trajectory optimization, time series analysis in signal
processing, econometrics and more.
Kalman filter is a recursive algorithm which uses time-series measurement over time, containing statistical noise and produce estimations of unknown variables.
For the one-day prediction Kalman filter can
be used, while for the long-term forecast a linear model is used where its main
features are Kalman predictors, infected rate relative to population,
time-depended features, and weather history and forecasting.
The one-day Kalman prediction is very accurate
and powerful while a longer period prediction is more challenging but provides
a future trend. Long term prediction does not guarantee full accuracy but
provides a fair estimation following the recent trend. The model should re-run daily
to gain better results.
The Center for
Systems Science and Engineering at Johns Hopkins University has developed an
interactive, web-based dashboard that tracks the status of COVID-19 around the
world. The resource provides a visualization of the location and number of
confirmed COVID-19 cases, deaths and recoveries for all affected countries.
The primary
data source for the tool is DXY, a Chinese platform that aggregates local media
and government reports to provide COVID-19 cumulative case totals in near
real-time at the province level in China and country level otherwise.
Additional data comes from Twitter feeds, online news services and direct
communication sent through the dashboard. Johns Hopkins then confirms the case
numbers with regional and local health departments. This kind of Data analytics
platform plays a pivotal role in addressing the coronavirus outbreak.
All data from
the dashboard is also freely available in the following GitHub repository.
One of AI’s core strengths when working on identifying and limiting the effects of virus outbreaks is its incredibly insistent nature. AI systems never tire, can sift through enormous amounts of data, and identify possible correlations and causations that humans can’t.
However, there are
limits to AI’s ability to both identify virus outbreaks and predict how
they will spread. Perhaps the best-known example comes from the neighboring
field of big data analytics. At its launch, Google Flu Trends was heralded
as a great leap forward in relation to identifying and estimating the spread of
the flu—until it underestimated the 2013 flu season by a whopping 140 percent
and was quietly put to rest. Poor data quality was identified as one of the
main reasons Google Flu Trends failed. Unreliable or faulty
data can wreak havoc on the prediction power of AI.
Bargunan is a Big Data Engineer and a programming enthusiast. His passion is to share his knowledge by writing his experiences about them. He believes “Gaining knowledge is the first step to wisdom and sharing it is the first step to humanity.”
Data is a crucial component for any organization to generate revenue and provide the best-in-class experience for their customers. Various studies have shown that 60% of the organizations fail to implement UI tools, which are heavily dependent on data-driven technologies because organizations spend millions on buying these tools but not investing in the right talent to achieve them. Understanding of data is the first stepping stone for any organization to be data-driven. I implemented various data solutions from inception to implementation, which helped organizations to derive data-driven decisions. After fifteen years of extensive experience across multiple data technologies and platform, I have developed numerous critical data frameworks which have benefited organizations to be data-driven. The first essential pillar is to build a cohesive and robust enterprise data team.
Data is a driver for any business intelligence, analytics, insights, marketing campaigns, UI applications, tools, and technologies. It’s crucial to understand why and what the business needs before deciding to invest in any data technologies. Today, organizations are leveraging data for executing campaigns and defining customer 360-degree views to provide personalized and OMNI-channel experience using data KPIs. There are unlimited data tools available, and it became difficult to pick the right one, which fits all the requirements for the business and delivers a perfect solution. It all goes back to find the right leader who has deep experience on both sides of the coin (Business and Technology). It’s hard to find such talent but not impossible, and this decides the success or failure of any data implementation projects.
About the Author:
Sankul is the Vice President of the Enterprise Data Team at PSCU. is a value-driven and business-oriented data and IT technology leader with a proven track record for building enterprise applications and data-driven platforms. He believes the current generation and future leaders should be focused and good listeners, as it helps to perceive and deliver solutions.
GAVS uses cookies to personalize and improve our reader experience. By pursuing your navigation on our website, you agree to our use of cookies as described in our cookie policy.Got itNo
