Tag Archives

10 Articles

Power BI Refresh scenarios against data storage solutions explained

by Jeroen ter Heerdt 0 Comments

A recurring theme with customers and partners is Power BI data refresh. More specifically, there is some confusion on what refresh scenario requires a Pro version and what can be done with the Free version of Power BI. I made the diagram below to help explain this. It shows the refresh scenarios against data storage solutions, such as SQL Azure, SQL in a virtual machine in Azure or SQL server on premises. I used these as examples, there are other options as well. I think the overall time carries over to other data storage solutions. The diagram shows the refresh that can be done using a Power BI Free account as orange and the refresh scenarios that need Power BI Pro as green lines. As shown in the diagram, if  you want to refresh against on-premises sources or a database running in a VM in Azure you will need a gateway and Power BI Pro. This applies not only to the creator of the report and schedule but also to every consumer. If you use PAAS solutions for data storage in Azure such as SQL Azure, it becomes a bit more difficult and it is really dependent on the type of refresh required. If you need a refresh cycle higher than once a day (either max 8 times per 24 hours or live) you will need Power BI Pro. If you just want to refresh against such as SQL Azure and once a day is enough you can do that using Power BI Free. Again, the license requirement carries over from author to viewer; if the author of the report requires Pro, then the viewers also need Pro.

Power BI Refresh scenarios against data storage solutions

Hope this helps. If you have any questions or feedback, please comment below!

Azure Machine Learning pricing explained

by Jeroen ter Heerdt 3 Comments

Many customers asked me questions on Azure Machine Learning (Microsoft’s fully managed machine learning and data mining solution) and more specifically on it’s pricing. In this post I will try to explain how the pricing works and what components you need to be aware of.

Azure Machine Learning is offered in two tiers: Free and Standard. The Free tier is obviously, well, free. It is however as you could expect limited compared to Standard. Differences are mostly in performance (multiple nodes for execution in standard vs. just one node in free) or storage (10 gb in free, unlimited in standard). There is no SLA for the free version, you cannot set up a production Web API to automate experiments in free and the staging web API is throttled.

For the standard tier, the following items need to be taken into consideration:

  • Seat; Azure ML has a monthly fee per seat, which translates to a user (mostly your data scientist) using the Azure ML web interface to develop and tune experiments. This price is per month per subscription/seat.
  • Studio usage; This is an hourly price for running experiments. You will pay this according to the number of hours your experiments run and thus claim computing resources.
  • API Usage; Azure ML allows you to bring an experiment online through the use of RESTful web services. This means you can automate score and training and have applications, websites, etc. use the experiment without human interference. With this you could do an automated credit scoring, recommendation or churn prediction directly from your app or website. In order to make this work you will need to create a web service in Azure ML (also called API). Azure ML charges per hour for compute used in an API that is production, so that is the fee you will need to pay per hour the web service / API is ‘online’ and usable. Also, you will need to pay per 1000 transactions. Transactions in this case are interactions with the API, such as one recommendation, one churn or one credit score.

 

Hope this clarifies a bit. Please refer to the official page linked above for more details and for the pricing details.

 

Reporting on your Azure subscription made easy using Power BI

by Jeroen ter Heerdt 2 Comments

Just released, an Azure Enterprise Subscription content pack for Power BI. Reporting on the usage of your Azure subscription does not get any easier!

Check it out: https://powerbi.microsoft.com/en-us/documentation/powerbi-content-pack-azure-enterprise/

First Look: Azure Data Catalog

by Jeroen ter Heerdt 0 Comments

The First Look series focusses on new products, recent announcements, previews or things I have not had the time to provide a first look at and serves as introduction to the subject. First look posts are fairly short and high level.

Azure Data Catalog is a service that is now in public preview that provides a one-stop access layer to data sources; it abstracts away specifics of accessing data that are dependent on where and how data is stored, such as server names, protocols, ports, etc. It includes easy to use search and publishing tools, so both business and IT can collaborate together on providing a general, easy to use data access layer to all employees.

For more info on Azure Data Catalog see: http://azure.microsoft.com/en-us/services/data-catalog/

Link to a one-slide Powerpoint to show the latest services in Azure

by Harry Tolsma 0 Comments

Use below link to download a very usefull slide which shows you all IAAS and PAAS services within Azure:

Overall services in Azure

Have fun,

Harry

Platform as a Service (PAAS) moving a rocket speed!

by Harry Tolsma 0 Comments

As an Enterprise Architect in an organization life has always been dynamic to say the least! It is your responsibility to keep up with the latest developments in ICT both in technique as in architecture. In the old days of on-premise only that was a big challenge. But with the Cloud as a integral part of your information systems it became even more complex.

But still… The Cloud was moving vm’s to Amazon or Microsoft. So architecturally not that complex. Identity & access off course, but that’s about it. Then came Platform as a service (PAAS). That was something completely different! Not moving vm’s to the Cloud, but move complete technical workloads to the Cloud like an ESB in the Cloud, Media Services, Federated identity, Storage, etc, etc..

This does impact your architecture!

A blazing 78 new PAAS services were introduced in 2014 within Azure. So it’s moving rocket fast! And to be fair: not only at Microsoft, also are other Cloud vendors moving into the PAAS area with new services.

What is the impact for you as Enterprise Architect?

In your normal day to day work you make choices based on software you can purchase and implement at your data center. But now you should at least ask yourself for every choice you have to make: Do I want to do this myself or shall I take this as a service from one of the Cloud vendors.

An example: Your organization wants to use Cloud services from multiple Cloud vendors but you want a single sign on experience for your users. Now you can buy a federated identity server, do research on all Cloud vendors on how to connect and then build the connections. But you can also use The Windows Azure Active Directory Federation Service (ADFS) from Microsoft with over 2600 Cloud vendors already pre-installed.

Second example: You have a new web application that you need to deploy. Again you can buy a few servers, install IIS, SQL Server, the application and install everything and schedule things like backup, patch management, storage, etc., etc. But you can also take a web-role to host the web-application, Azure SQL database to host you data and let Microsoft worry about backup’s, 3 replica’s for DR, patching the server, etc.., etc.

So my message to all you Enterprise Architects out there: Examine carefully the PAAS offerings from the Cloud vendors before making expensive buy decisions. My recommendations to checkout:

Azure Service Bus, Azure Machine Learning, WAAS, BizTalk Services and Azure SQL Database. Next blog-post I will dig deeper on Azure SQL Database.

First Look: Azure Data Factory

by Jeroen ter Heerdt 0 Comments

This is the first post of my new first look series. This series focusses on new products, recent announcements, previews or things I have not had the time to provide a first look at and serves as introduction to the subject.
First look posts are fairly short and high level.

Today in first look: Azure Data Factory. This service was only recently announced as is available to all Azure customers in preview mode. To get a hold of it make sure you open the Azure preview portal. In your classic Azure portal click on your email address in the top right and choose ‘Switch to new portal’ or go directly to https://portal.azure.com.

So what is Azure Data Factory? I may be downplaying it a bit, but essentially Data Factory gives you ETL in the cloud. It connects to both on premises as well as cloud data stores and enables you to read data from the stores, do transformations and publish data in stores, while at the same time providing rich analytics on how the data flow is doing. The paradigm here is a factory floor: pieces of data enter the factory floor as raw materials, they undergo some treatment (transformations) and go out the door at the other end of the floor as finished product. The canvas of Data Factory closely resembles this floor and shows an assembly line for data. Here is very simple example, which retrieves data in hourly batches from a table in Blob Storage and stores it in a SQL Azure table:

 

More info is available on http://azure.microsoft.com/en-us/documentation/articles/data-factory-introduction/.

More to come!

 

Migrating your local virtual machine to Azure

by Jeroen ter Heerdt 0 Comments

If you have a local VM that you would like to migrate to Azure, here is how to do it: http://www.windowsazure.com/en-us/documentation/articles/virtual-machines-create-upload-vhd-windows-server/

Enjoy

Managing Azure Virtual Machines using PowerShell

by Jeroen ter Heerdt 0 Comments

I use Azure for my demo machines and needed a way to easily shut all VMs down and also to start my BI demo machines. What better way to do this other than with PowerShell!

In order to be able to talk to Azure with Powershell you will need to install Windows Azure Powershell (download link: http://www.windowsazure.com/en-us/downloads/ ).

Also you will need to get your Azure Publish Settings file which is specific to your subscription. You can get it from the Azure portal once you are logged in: https://windows.azure.com/download/publishprofile.aspx .

Let’s start with stopping all VMs. Start the PowerShell ISE (search for powershell_ise on Windows 8) and create the following script:

Make sure to enter the path to where you stored your publish profile that you have downloaded.

Basically what this script does is iterate over all your subscriptions if you have more than 1 and look for VMs that are in a running state using get-AzureVM. Then for each VM that is running it will echo its name and then stop the VM using stop-AzureVM.

Save the script and then you can just run it and all of your VMs will be turned off. Pretty easy huh?

For my BI demos I use a maximum of four VMs and I made another script that starts them in the correct other (first the domain controller, then the SQL server and then finally the two SharePoint servers I need):

This script defines a function that wraps a check if the VM is already running and otherwise starts it. The bottom part of this script uses that function to specify which VMs to start in which order. I replaced the original names for security reasons.

This saves a lot of time. It saves me from logging into the Azure portal and starting / stopping each VM by hand. The best part is I can let this run in the background while presenting and nobody sees it J


Working with Azure and HDInsight from SSIS

by Jeroen ter Heerdt 0 Comments

A while ago a whitepaper was published on how to work with Azure and HDInsight (Hadoop on Azure) from SSIS. In that whitepaper some code samples were given. That code is also available here (including some components): http://code.msdn.microsoft.com/SSIS-Packages-Sample-for-2ffd9c32 . After you have downloaded the zip make sure you have Visual Studio installed, start the Developer Command for Visual Studio as Administrator and run the ‘deploy_SSIS_packages_and_components.bat’ file which is included in the zip. This will install some of the DLLs included into the GAC. Then, you can open the solution in Visual Studio.

The solution includes the following sample SSIS packages:

  • PigSqoopPackage: shows how to work with Pig and SQOOP tasks
  • HadoopJobAutomation: shows how to start jobs on Hadoop and how to consume results
  • ComplexSourceDestination: shows how to get data from Azure Blog Storage and save the results into various targets, including Azure Blob Storage.
  • BlobSourceTestPackage: sample package showing how to read data from Azure Blob Storage.
  • BlobDestinationTestPackage: sample package showing how to write data to Azure Blob Storage.

The first two packages essentially contain some script tasks with complete samples on how to work with Piq, SQOOP and Hadoop jobs respectively. The other packages use the components provided and provide a quick start on getting data from Azure Blob Storage and getting data into Azure Blog Storage using SSIS.

%d bloggers like this: