Recent articles filed under Machine Learning

Did you know DVC doesn't handle large datasets? Neither did we and here's how we fixed it

Created 1/13/2023 9:37:46 AM
Filed in Machine Learning

We hit a nice roadblock yesterday while working on a dataset with 5 million samples. It took us 8 hours to upload the dataset and 11 hours to download and unpack it. We learned some valuable lessons there that I’ll discuss in this post so you don’t have to run into the same issues.

How to configure memory limits in WSL2

Created 1/5/2023 8:01:11 AM
Filed in Machine Learning

One of the downsides to building machine learning models is that you need quite a bit of power in your machine. I've just upgraded my home desktop that I use for side projects and experiments to 64GB of memory because it was clearly no longer up to the task. However, when I booted up my WSL linux distro, it couldn't get more than 32GB of memory. Time to figure out how to fix that! In this quick post I'm showing you how you can limit or increase the memory available to your WSL2 distributions. Let's go!

How to build scalable workflows with Prefect and Ray

Created 12/23/2022 7:38:36 AM
Filed in Machine Learning

In the past weeks we've been working on a new computer vision project at Aigency. As part of the project we have to process images using OpenCV image filters. This is a rather tedious process that takes a lot of time, so we made a nice workflow for it. In this blog post I'll show you how we used Prefect and Ray to build a scalable image processing workflow.

How to set up CUDA and PyTorch in Ubuntu 22.04 on WSL2

Created 11/2/2022 6:24:44 PM
Filed in Machine Learning

Working with machine-learning code in Windows can be quite a hassle. Often I run into problems with my tools not working properly on Windows. Recently, I even decided to convert my laptop to Ubuntu 22.04 for the time being. Which is really nice for data sciency stuff, but not great for my Powerpoint and Excel since LibreOffice sucks big time. Needless to say, I needed another solution. Especially for my deep learning setup. In this quick post I'll show you how I configured my Ubuntu 22.04 installation on Windows 11 to run CUDA and PyTorch.

Training machine-learning models with DVC

Created 10/17/2022 8:58:56 AM
Filed in Machine Learning

I've shown how to use DVC to manage data for your machine-learning project on my blog before. Today it's time to take things to the next level. Let's build a pipeline to train a machine-learning model with DVC.

Managing machine-learning datasets with DVC

Created 10/14/2022 9:59:21 AM
Filed in Machine Learning

DVC is a data versioning tool that is designed to make it easier to work with data for machine-learning projects. It integrates with git so that you can easily keep track of your data files and track changes to them over time. It also allows you to easily share data files between different projects and collaborators. In this post I'll show you how you can start using DVC in your ML project.

How to debug Airflow DAGs in VSCode

Created 9/10/2022 6:02:00 PM
Filed in Machine Learning

After last week's success setting up Apache Airflow I figured it would be nice to have a development setup so I can debug my own stuff. In this post, we'll look at how you can use VSCode to setup debugging for your Airflow DAGs.

How to deploy Airflow with Kubernetes and GIT

Created 9/8/2022 7:04:31 PM
Filed in Machine Learning

When I started working on MLOps a couple of years ago I ran into Azure Machine Learning first, because that's what the community around me was using. Turns out, it's not the only product out there. I already knew this, but I Ignored other products because we were happy with Azure ML at the time. This may sound like I'm no longer happy with Azure ML and want to move away. But that's not what this is about. I think you should always explore other technology options from time to time. That's why I'm trying out Airflow on Kubernetes.

Build the ideal python development environment in 2021

Created 7/23/2021 1:37:17 PM
Filed in Machine Learning

So you want to start building a python project, what should you use? I've talked about this before numerous times. Many things have changed for me over the past few years. Here's my recommendation for 2021.

Break open your machine learning model with plsexplain

Created 7/1/2021 5:33:17 AM
Filed in Machine Learning

I built a new tool called plsexplain that asks the computer: Please explain this machine learning model to me?! This tool will make it a lot easier to break open the black box of your machine learning model.

Fairlearn: The secret to teaching your models to play fair

Created 5/19/2020 5:00:00 PM
Filed in Machine Learning

Microsoft just announced a new suite of tools around Responsible AI during the Microsoft Build conference. One of the tools that they announced is fairlearn. A new python package that will help you build fair models. Let's see the secret sauce to building fair models.

ML.NET Tutorial: Getting started with ML.NET

Created 8/2/2019 7:25:48 AM
Filed in Machine Learning

Microsoft built a new machine learning library a while ago. It allows C# engineers that want to take their machine learning to a higher level to build custom models rather than using the standard cogn