Delete old document versions from OneDrive for Business

I’m using OneDrive included in my Office 365 subscription to store a copy of my local backup in the cloud. My subscription provides 1 TB cloud storage for every user, which is quite a decent amount of storage space, at least for my needs, so I was really surprised when I received the warning that my OneDrive is full. How can that happen, my backup data is only 0.5 TB locally, how can that consume twice as much space in OneDrive?

What takes up so much storage space?

I sync tons of data to OneDrive, so first I had to understand what actually consumes the most storage space. OneDrive by default does not show the size of folders, but here is the trick to view that:

Click the Gear on the top right corner, than click Site settings on the sidebar:

Site-Settings-menu

If you ever administered SharePoint, the list of menu items will be familiar to you: your OneDrive for Business storage is actually a SharePoint site collection. Click Storage Metrics in the right column:

Site-Settings

Here you can see all your files and even the system folders with a nice inline bar, so you can immediately see which folder consumes the most space. Actually all your data is in the Documents folder:

Storage-Metrics

By clicking the folder names you can dive deeper and see subfolders and files. Note that the Total Size includes the following:

  • Deleted files in Recycle Bin
  • All versions of the file
  • Metadata associated with the file

To check the versions you can click the Version History link on file level:

File-List

Here you can see that OneDrive for Business does have file versioning enabled by default, so when you overwrite a file, the previous version remains there and consumes storage space from your quota. By clicking the Delete All Versions link you can get rid of the old versions:

Delete-All-Versions

Unfortunately you have to do it file-by-file, there is no UI option to delete the old versions of all files from your subscription.

Recycle Bin

As a first step I wanted to check whether my Recycle Bin contains any files, because if it does, it counts into my quota.

So I opened my OneDrive in the browser and clicked Recycle bin in the left menu:

OneDrive-Recycle-Bin

Unfortunately it does not show how big are those files in the Recycle Bin, but clicking Return to classic OneDrive in the bottom left corner you can get a different view, where you can sort the file by Size, and Empty the Recycle Bin with a single click:

Recycle-Bin-file-size-list

 

Do you think you are done? Not yet! Recycle bin is there to protect you from accidental file deletion, but there is a second-stage recycle bin to protect you against accidental recycle bin emptying:

Recycle-Bin-empty

So click the second-stage recycle bin link (which is actually the admin level recycle bin on the site collection) to see that your files are still there:

Second-Stage-Recycle-Bin

Click the checkbox at the top of the first column to select all files, then click Delete Selection. Bad news: the list is paged, and you have to do it page by page 😦

Disabling document versioning in OneDrive for Business

Because your OneDrive for Business is actually a SharePoint site collection, and all your files are in the Documents folder which is actually a SharePoint document library, you can disable versioning for all your documents by disabling versioning in that single document library. To find that configuration option first click Return to classic OneDrive in the bottom left corner, then the Gear on the top right corner, and then click Ribbon in the right sidebar to view the ribbon:

Enable-Ribbon

Expand the ribbon by clicking the Library tab, and then click Library Settings towards the right end of the ribbon:

RIbbon

The settings page of the document library contains several useful option and information:

Documents-Settings

First, if you have more than 5000 files in your OneDrive (like I do), a List view threshold warning is shown on this page. This will be important later, because as the description says, you cannot do operations on that many files at once.

Second, the Versioning settings link leads to the page where you can disable versioning, or limit the number of versions retained:

Versioining-Settings

Both option can help you reduce the storage space needed for your files. Note however that turning off versioning here does not delete the existing versions of your files!

Delete old document versions of all files

Unfortunately there is no option on the UI to delete the old versions of all your documents with a single click, but thankfully SharePoint has an API, so you can do it programmatically.

If you have multifactor authentication enabled (you do, right?) then the first step is to generate an application password. To do that click your avatar in the upper right corner, then click the My account link, or navigate directly to https://portal.office.com/account/.

On the left menu click Security & privacy, then on the middle pane click Additional security verification. It does not look like a menu item, but it actually is, and expands the pane to show new items:

Security-and-privacy

Click the Create and manage app passwords link and use that page to generate a unique password for your app.

The next step is to download the SharePoint Online SDK which installs a set of DLLs into the C:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\ISAPI folder. You can use these DLLs with the PowerShell scripts published by Arleta Wanat to the TechNet Script Center:

These are very handy scripts, note however that they have a limitation: they cannot deal with more than 5000 files and you can receive the following exception:

“Microsoft.SharePoint.Client.ServerException: The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.”

Thankfully CAML has the power to limit the number of returned items using RowLimit and add paging, however it does not always work if you want to run a recursive query. But the option to run a separate CAML query to each and every folder is working.

I implemented my final solution in a good old .NET console application using the SharePoint client-side object model. The application executes the following steps:

  1. Connects to your OneDrive for Business or SharePoint site collection.
  2. Finds the Documents document library.
  3. Iterates through the specified subfolder paths in the document library.
  4. It runs CAML queries in every folder to retrieve the documents (files). In a single query maximum 100 documents are retrieved, and the query is executed again and again until all documents are processed.
  5. If a document has multiple versions, they are deleted.

You can find the code in Github and customize it according to your needs: https://github.com/balassy/OneDriveVersionCleaner

 

Technorati-címkék: ,,,
Advertisements

Show me your imports, and I’ll tell who you are

Well, maybe not, but your imports may indicate, how maintainable is your code.

Have you ever written something like this:

import { SomeClass } from 'somelib';
...
const s = new SomeClass();

Or this:

import * as someFunc from 'somelib';
...
const s = someFunc();

Looks familiar, right? I’ve written a lot of lines like these, mainly because the README in every library, package recommends them. But there are (at least) two problems with this code why I am trying to avoid them.

Dependency management

If you unscrupulously follow this pattern, you will write the same line of code hundred (or more) times within a single application. Why? Because we love copy-paste. Easy, convenient, and it works, right?

However in this way it will be very difficult to determine what your app exactly needs, what it depends on, what are its dependencies. What one can easily see from your code (only after using the “Find in All Files” feature of your favorite editor), is that your code references to an external package in countless places, so it needs that package, your app depends on it.  

But reality is different: your application logic does not need a package, but a functionality, a logic, which happens to be implemented in a third party package. But it’s not what your code shows at the first sight.

Here’s a more specific example with the popular lodash package (*):

var _ = require('lodash');

This package provides a number of utility functions, and it is pretty much impossible, that your app needs all of them. How can you find out what exactly you are using? Search all your files for “_”.

But why is that interesting?

For example, because you may want to replace the referenced package with something else. In the open source world, packages come and go, today this was the best, tomorrow that will be the one. Because the old package contains a bug, that will never be fixed, or because only the new package is compatible with the latest Node version that other parts of your app requires. Or simply because you want to keep your codebase fresh, and you want to use only dependencies with active community.

You can be sure that you will not find another library or package, that is API-compatible with the old one. You need one that is compatible on functionality level, which you can only find out if you know exactly what functionality are you referring to.

When you have that, you just have to change the references in as few places as possible in your code.

This leads to the solution: make sure, that you reference third party packages only once in your code, so wrap them into classes that publish only the functionalities your app depends on. This method can also make your code more readable, for example, instead of "sha256()", you can find a much better function name, right?

This method works for me very well for a long time, though I have to note I don’t follow it fanatically, because it will be too much overhead for framework level libs (e.g. Angular).

Testability

If your class uses “import” to refer to another one, unit testing that class will be a nightmare, because you will have to hack the file loader logic to mock the class (e.g. Mocking Request-Promise).

The solution is not new: DI. Not only does it make testing easier, but typically you just have to take a look at the constructor, and you will see on what other classes this one depends on.

For example, in TypeScript, we often import interfaces just to have type description, which can make import rows very verbose, and often you cannot determine at the quick glance which import brings in functionality and which one is there for type information only. DI also helps in this, if we try to use “import” only for type descriptions, and DI for everything else.

Is this something new?

Not at all. Really. Yet these codes continue to live, even if they are hard to maintain. READMEs, tutorials, articles, blogs will always strive for simplicity, you will find examples of them that help you understand the content and get started. Their purpose is not to blindly follow them in a more complex application where maintenance and sustainability is as important as simplicity. When you write such a line next time, think about what you’re really bringing into your codebase. Do you really gain some time with it on the long run?

programming-is-thinking_thumb

 

*: The example is not perfect, because lodash provides more fine grained imports, however it is not well known or widely used.

 

Technorati-címkék: ,,

Printing five-dimensional objects with a 3D printer

On this day exactly five years ago, the Volume 49, Issue 1 of the British Journal of Psychology was made available online, in which Lionel Penrose and his son, Roger Penrose published an article titled Impossible Objects, about objects that physically cannot exist. Probably the most well-known among those objects is the Penrose-triangle:

5d-penrose-triangle_thumb[1]

The original article was published much earlier, in 1958, and in the 50 years that passed since then, we learned a lot about our world. With our current knowledge we can confidently say, that the Penroses were wrong, these objects can exist, and we can create them using the modern 3D printing technology.

As a first step, we need a model we will print. The first problem is that traditional graphic and modeling softwares can work only with three dimensions, and to print these objects, you will need more. So you can forget SketchUP, Blender, AutoCAD, Maya, Tinkercad etc., because they understand only the 3D world.

The only possible option (at least I’m not aware of any other) is to describe your model in code, instead of drawing it. OpenSCAD is a wonderful tool for it! Even if you have only a little affinity to programming, I absolutely recommend you to use OpenSCAD to define your 3D models. Because we are working with code here, we are not limited by the three-dimensional space, and we can use transformations to move our objects to additional dimensions. The most important is the to5d() method here, that you can find in the Transformations group in the OpenSCAD CheatSheet:

5d-openscad_thumb[1]

After we have the model, the next step is slicing it. I mostly use Cura to slice my STL files, because its UI is very intuitive, and not too crowded with controls. Probably this is the reason why many controls are available only through the Expert config dialog:

5d-cura-expert-config_thumb[1]

Unfortunately you cannot find five-dimensional printing options even here by default, but if you look thoroughly, you can notice that while most labels are nicely vertically distributed, the labels in the Black Magic group are nearly touching each other. This is because if you enable both options there, a third one, called Enable more dimensions is displayed in that narrow space. Let’s enable that too:

5d-cura-black-magic-settings_thumb

With that option enabled, Cura can seamlessly open the 5D STL file created by OpenSCAD, and we can print it as usual. This is my printed Penrose-triangle:

5d-triangle_thumb[1]

Another photo with a classic 3D object:

5d-triangle-with-pencil_thumb[1]

I made these photos with my phone without flash, and the curvy edges of the objects are the result of the limited capabilities of my entry-level printer.

After I successfully printed my first “impossible” object, I wanted to print the triangle of the Swedish graphic artist, Oscar Reutersvärd (source: Wikipedia):

5d-reutersvard-triangle_thumb

The OpenSCAD model is much simpler here, I only had to create 9 cubes. After that I could open the model in Cura:

5d-cura-preview_thumb[1]

You can see a lighter region on the preview image at the second cube on the right from the bottom. To be honest I cannot explain what caused it, but thankfully it is just an error in rendering, I could successfully print this too. This is the result:

5d-cubes_thumb[1]

When we are talking about impossible objects, we can not talk about M.C. Escher’s famous picture with the endless staircase. This is how it looks in Cura:

5d-cura-stairs_thumb[1]

After my printer completes the print, I will upload the photo here.

As you can see from these examples, 3D printers can be used to print more than three-dimensional objects, so don’t get stuck into 3D!

What will you print?

Technorati-címkék:

Favorites and bookmarklets in Microsoft Edge

I use Microsoft Edge as my primary browser since Windows 10 was released, and I am really satisfied with it. I don’t miss the plugins, but saving a bookmarklet was trickier than I thought.

Let’s take the Pinterest bookmarklet, the Pin It Button as an example. In every other browser you can navigate to the webpage of the bookmarklet, and drag the button which contains the JavaScript code to your favorites. For security reasons this does not work in Edge, but you can fix it.

The case was much simpler before the major update in November 2015, because the favorites were stored in the file system, in this folder:

C:\Users\{FELHASZNÁLÓNÉV}\AppData\Local\Packages\Microsoft.MicrosoftEdge_8wekyb3d8bbwe\AC\MicrosoftEdge\User\Default\Favorites

According to the sources all over the Internet, now they sit in the spartan.edb ESE database here:

C:\Users\{FELHASZNÁLÓNÉV}\AppData\Local\Packages\Microsoft.MicrosoftEdge_8wekyb3d8bbwe\AC\MicrosoftEdge\User\Default\DataStore\Data\nouser1\120712-0049\DBStore

Because this is a database, you need special tools and special care to edit it, but luckily it seems that the browser also uses registry. I managed to find my favorites in this location in the registry:

HKEY_CLASSES_ROOT\Local Settings\Software\Microsoft\Windows\CurrentVersion\AppContainer\Storage\microsoft.microsoftedge_8wekyb3d8bbwe\MicrosoftEdge\FavOrder\FavBarCache

Based on this, I could add the Pinterest bookmarklet to my browser with these steps:

  1. I navigated to the page of the Pinterest bookmarklet, right clicked the “Pin It” button, and then selected the Copy Link item from the menu to copy the JavaScript code of the bookmarklet to the clipboard.
  2. After this I added this page to my favorites just to have the Pinterest icon for my bookmarklet.
  3. I started Regedit, and navigated to the path above.
  4. The folder with the largest number contains the data for the favorite item I created on Step 2. I pasted the JavaScript code from the clipboard to the url key.
  5. I restarted the browser.

bookmarklet-regedit

This is totally unofficial, not supported and not a perfect solution, but works on my machine.

 

Chocolatey and security

I decided to use Chocolatey to install applications to my freshly installed Windows 10 machine. My original idea was to use OneGet, which is the new package-manager manager introduced in Windows 10, and which has a preview Chocolatey provider, however I didn’t have much success with it, so I stepped back and used Chocolatey directly.

The first step is to install Chocolatey, which is very simple, just run the 1-line script from the chocolatey.org homepage in an admin command prompt:

C:\> @powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((new-object net.webclient).DownloadString(‘https://chocolatey.org/install.ps1’))" && SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin

With this single command you actually do three things:

  1. You download a PowerShell script.
  2. You run the downloaded script with administrative privileges.
  3. You extend your PATH environment variable.

I don’t know about you, but step 2 freaks me out. And this can be a good time to take a deep breath and think through what you are going to do: you will install applications from unknown source to your machine! When you execute a command like

choco install adobereader

you have no idea what it will download and install to your computer.

So what can you do?

First, install only those choco packages that are approved by moderators. Moderation is a manual process, and it may have human errors, but it is a validation after all. In the detail page of a package that was approved by a moderator, for example the Adobe Reader package, you can see this in a green box:

This package was approved by moderator gep13 on 6/11/2015.

If a package was not checked by a moderator, for example Notepad2, you can see this in a red box:

This package was submitted prior to moderation and has not been approved. While it is likely safe for you, there is more risk involved.

If you already opened the detail page of a package, you better read everything you can find there. For example in the 7-zip package page you can find this warning:

NOTE: The installer for 7-Zip is known to close the explorer process. This means you may lose current work.

Here you can also find useful options too, for example the Firefox package allows you to specify the language of the application to install:

choco install Firefox -packageParameters "l=en-US"

If you scroll down, you can find references in the comments which may make you choose not to install a certain package. For example OpenCandy is mentioned in the comments of the CDBurnerXP package, and you can probably also recall installers that install unwanted software if you just blindly go through them with next-next-finish.

In the middle of the page you can find the installer PowerShell script as well, which might be worth to take a look at, because there you can see, what EXE or MSI is downloaded and from what server. In case of the Adobe Reader package this script is only 6 lines, you can clearly see the URL in the middle, and you can very easily understand what is actually happening. One cannot say the same about the 117-line script of the Firefox package, or the script of the Node.js package which is only 1 line, but refers to two other packages.

In summary, I don’t feel Chocolatey can be used securely, it is all about trust. You can do these manual checks, you can update your operating system, install antivirus and antimalware, deny access to unwanted hosts, but at the end of the day you will run code from an unknown source, which – at least from security perspective – doesn’t seem to be a good idea.

I installed these packages, and they didn’t burn down the house (yet):

adobereader
7zip
emet
fiddler
filezilla
firefox -packageParameters "l=en-US"
gitextensions
google-chrome-x64
join.me
keepass
nodejs
paint.net
silverlight
skype
sysinternals
vlc

So what do you think, do you use similar tools, are you brave enough to use Chocolatey, and if yes, what other packages do you install?

 

Technorati-címkék: ,,

Use Bitlocker without TPM

Contrary to the popular belief, you can use Windows’ built-in Bitlocker to encrypt your hard disk content, even if you don’t have a TPM chip in your computer. You can easily encrypt your data disks; you just have to enter a password, and you have the option to save your recovery key to file, a USB drive or even to the cloud. However when you try to encrypt your OS volume with Bitlocker, you will see the following error message:

This device can’t use a Trusted Platform Module. Your administrator must set the “Allow BitLocker without a compatible TPM” option in the “Require additional authentication at startup” policy for OS volumes.

bitlocker-1-tpm-error_thumb[1]

It is quite a good error message, because it not only states what the problem is, but also helps you to recover from it. If it would say exactly where you can find that setting, it would be perfect!

If you search for the word “policy”, you will find Local Security Policy, but it is not what you really want. What you need is the Group Policy Object Editor, even if your computer is not domain joined.

Start a Microsoft Management Console (mmc), and add the Group Policy Object Editor snap-in (click for the full image):

bitlocker-2-mmc-add-snapin_thumb[1]

Then within then Local Computer Policy –> Computer Configuration –> Administrative Templates –> Windows Components –> BitLocker Drive Encryption –> Operating System Drives branch you can find the setting the error message referred to:

bitlocker-3-policy-path_thumb[1]

Open the setting, then first select Enabled, and then click the Allow Bitlocker without a compatible TPM checkbox below:

bitlocker-4-policy-setting_thumb[1]

After you have closed all windows, you have to refresh your security policy, which you can do without restarting your computer by running gpupdate from the command prompt:

bitlocker-5-gpupdate_thumb

Now you can encrypt your OS volume just as you did with your data disks.

 

Technorati-címkék: ,,

Windows 10 install: UEFI, secure boot, USB, GPT, error

I’ve tried to install Windows 10, so I have downloaded the ISO from MSDN, and used the Windows USB/DVD Download Tool to write it to a pendrive. However, my computer refused to recognize the installation media, so I didn’t have to option to boot from it. In my BIOS the boot options were set to UEFI boot ON, secure boot ON, which worked well for the previous Windows 8.1, but now these settings caused the problem. If I changed that setting to Legacy boot ON, secure boot OFF, the boot from USB option appeared, and the Windows installer successfully started. Unfortunately this is not the happy end of the story, because the installer later stopped with this error:

Windows cannot be installed to this disk. The selected disk is of the GPT partition style.

windows-install-gpt-error

 

After some googling I’ve found several methods to convert a GPT partition to MBR (with loosing all data on the whole disk, or by using a 3rd party boot CD), but fortunately there is a much easier method.

 

The much easier method

The real issue is that UEFI boot does not work with NTFS pendrives (at least not in my machine), so the solution is to

use a FAT32 pendrive.

Unfortunately the Windows 7 USB/DVD Download Tool always reformats the pendrive to NTFS, even if it was formatted previously, so you need another tool to prepare the installer pendrive.

You can use diskpart for example, which is a built-in command line tool in Windows.

Let’s start it:

diskpart

Get a list of the available drives:

list disk

You will see the pendrive in the list (because hopefully you previously inserted it), you can recognize it from its size. Tell the tool that you want to work with that disk:

select disk 2 (use the correct number instead of 2)

Remove every content from the disk (you will lose your existing data on the pendrive!):

clean

Create a new FAT32 partition:

create partition primary
select partition 1
active
format quick fs=fat32
assign

Now you can quit from diskpart:

exit

The last step is to copy the installer to the pendrive. First, mount to ISO, and copy its content to the disk, for example with xcopy (in this example D: is the mounted ISO drive, F: is the target pendrive):

xcopy d:\* f:\ /s /e

With that your BIOS will hopefully recognize the pendrive, so you will be able to boot from it, and you will not have any problem with the GPT partition, even if UEFI and secure boot is turned on.

 

Technorati-címkék: ,,