Azure Databricks

-

 Why Azure Databricks? 



Since Azure Databricks is a cloud based service, it has several advantages over traditional Spark Clusters. Let us look at the benefits of using Azure Databricks.

>Optimized Spark Engine: Data processing with Auto-scaling and Spark optimized for up to 50x performance gains.

>Machine Learning: Pre-configured environments with frameworks such as PyTorch, TensorFlow and sci-kit learn installed.

>Mlflow: Track and share experiments, reproduce runs and mange models collaboratively from a central repository.

>Choice of Language: Use your preferred language, including Python, Scala, R, Spark SQL and .Net -- whether you use serverless of provisioned compute resources.



>Collaborative Notebooks: Quickly access and explore data, find and share new insights and build models collaboratively with the language  and tool of your choice.

>Delta Lake: Bring data reliability and scalability to your existing data lake with an open source transactional storage layer design for full data life cycle.

>Integration with Azure Services: Complete your end to end analysis and machine learning solution with deep integration with Azure services such as Azure Data Factory, Azure Data Lake Storage, Azure Machine Learning and Power BI.

>Interactive Workspaces: Easy and seamless coordination between Data Analysts, Data Scientist, Data Engineers and Business Analysts to ensure smooth collaboration.

>Enterprise Grade Security: The native security provided by Microsoft Azure ensure protection of data within storage services and private workspaces.

>Production Ready: Easily run, implement and monitor your heavy data oriented jobs and job related statistics.


Databricks Utilities:

Databircks Utilities or DBUtils help us perform a variety of powerful tasks which include efficient object storage, chaining notebooks together and working with secrets.

All DBUtils are available for notebooks of the following languages: Python, Scala and R

Some sample installation commands in Python NoteBook:

dbuitls.library.installPyPi("torch")

dbutils.library.installPyPi("scikit.learn", version="1.19.1")

dbutils.library.installPyPi("azureml.sdk", estras="databrics")

dbutils.library.restrartPython()

None: DBUtils are not supported outside notebooks


Integrating Azure Databricks with Azure Blob Storage:









Comments

Post a Comment

Popular posts from this blog

Cloud Computing in simple

-
Cloud Computing =   Infrastructure as a Service (IaaS) + Platform as a Service (PaaS) + Software as a Service (SaaS) Cloud computing, also known as Cloud Technology, Cloud Applications or Cloud-based services. Cloud Computing is the practice of using a network of remote servers hosted on the internet to store, manage and process data, rather than a local server or a personal computer. Examples of cloud-based services include customer self-scheduling, data storage, online payment, and accounting software Different Type of Clouds Infrastructure as a Service (IaaS)  – Allows users with the services like Virtual servers, Virtual storage, Physical Database, Virtual Network, Physical Network. E.g. Google Storage ,  IBM Cloud Computing ,  Nimbus Platform as a Service (PaaS)  – Provides computational resources via a platform upon which applications and services can be developed and hosted. E.g. Databases, Data Stores,  Force.com ,  Google App Engine ,  Windows Azure  (Platform). Soft
Read more

How to Write an Effective Design Document

-
Image
How to Write an Effective Design Document Why write a design document? A design document is a way for you to communicate to others what your design decisions are and why your decisions are good decisions. Don’t worry if your design is not UML compliant and don’t worry if you didn’t use a special modeling tool to create it. The biggest factor that determines if your design document is good is whether or not it clearly explains your intentions. This presents a problem, however. In order to convey design decisions, you have to consider the audience that you are writing for. A peer developer will understand why a well-crafted class abstraction is a good design, however your manager will probably not. Because your peer developers and your manager have different concepts of what makes a design good, there is a need for two design documents; one for peer developers and one for managers. Each document serves a different and equally valuable purpose as you begin your project development. I
Read more

Bookmark

-
Good Blog about Architecture & design Patterns:   http://martinfowler.com/bliki/ http://www.sharepoint-videos.com/ http://go.cloudshare.com/Linkedin_ProPlus_SharePoint.html A Technical Q/A site: http://www.bookmarkresource.com http://blog.krisvandermast.­ com/ASPNET20PageLifeCycle.aspx http://blogs.vertigosoftware.­ com/shannonf/archive/2005/08/­ 07/1355.aspx http://john-sheehan.com/blog/­ net-cheat-sheets/ http://www.eggheadcafe.com/­ articles/20051227.asp http://duartes.org/gustavo/­ articles/Asp.net-Runtime-­ Cheat-Sheet-HttpRequest-­ HttpRuntime.aspx http://www.addedbytes.com/­ cheat-sheets/javascript-cheat- ­sheet/ http://www.antsmagazine.com/­ 2009/04/100-useful-cheat-­ sheets-for-web-designers-­ developers/ Nice link abt .Net 3.5: http://www.simple-talk.com/ dotnet/.net-framework/.net-3. 5-language-enhancements/ nice articles about JQuery. http://encosia.com/2008/05/29/ using-jquery-to-directly-call- aspnet-ajax-page-methods/ http://encosia.co
Read more