11/30/17

Download all your email information using this Python 3.6 program

Python 3.6 program to download email information


I have been collecting my emails in gmail for years and need to trim them to get more space. The were taking up over 5 Gigs and there are over 100,000 of them.

I looked for tools to do this, and the best I have found is thunderbird, but it did not satisfy me.

So, I decided to write a program to downlaod all the email information except the actual attachments to then later write code to sift though the emails and get me some analytics so I can trim batches of emails that meet arbitrary criteria that I could control.

I looked at using the python imap and email modules for this. But when putting together examples from the web, I ran into a lot of issues:


  1. speed, most examples only fetched one email at a time, when imap supports doing batches in mone fetch
  2. character encoding errors, especially with emails from years ago
  3. lots of other details.


We I got the code working and it downloads my 100,000 emails in about 40 minutes.

I then save the data for analysis in to either a json file or a csv file.

Here are the fields in the files:

n: number of email from 1 to n in order fetched
From: information from email header
To: information from email header
Subject: information from email header
Date: information from email header
Received: information from email header (useful for senders ip address or compute domail)
Rfc822msgid: unique message id (for gmail, you can just paste this into find box to pull up that email)
Size: total message size including attachments
uid: imap unique id to fetch same email
Attachments: python list format of filenames of attachments
text/plain:  plan text if in email body
text/html: if the email does not have plain text, then it will try to fill in the html text of the body

You just edit the code and set the username and password variables.
You must also set the ouput_filename variable to the file path and name of where you want the output file, do not add an extension, that is indicated in the next variable: output_type as either json or csv.

This program works for gmail, but you must go to gmail settings and enable imap.
This program has not been tested with other email services, but by adjusting the imapAddress, and if needed add the port number to the call (change bold number):

ms = imaplib.IMAP4_SSL(imapAddress, port=993 ) # open imap session ms


Enjoy.
Source code:  download_emails.py

10/27/17

Installing Ubuntu 17.10 Linux into VirtualBox on an iMac

Installing Ubuntu 17.10 Linux into VirtualBox on an iMac
(OS Sierra) with Retina display resolution:

This article shows the steps to setup Linux in a virtual environment on a mac with a very high resolution display with high DPI.

Associated video: YouTube: Installing Ubuntu 17 on Mac in VirtualBox
and Mac Virtual Box Ubuntu 17 adjusting to full HiDPI

It covers the following:
  • Downloading Ubuntu 17
  • Downloading VirtualBox and installing it
  • Installing Ubuntu into VirtualBox
  • Setting up VirtualBox Guest additions
  • Fixing display flickering
  • Settings for Ubuntu and VirtualBox to take full advantage of a high DPI display
  • Some recommended settings
There are some issues that needed to be solved that were tricky to find the solutions to:

Downloading VirtualBox and installing it:

download from: 
   http://download.virtualbox.org/virtualbox/5.2.0/VirtualBox-5.2.0-118431-OSX.dmg

open downloaded dmg disk image:


Double click on #1, and follow prompts and accept defaults.

Once installed, use Spotlight or Launch Pad to find VirtualBox and open it.

Installing Ubuntu into VirtualBox



Click the New button in upper left in VirtualBox window






Add a name of your choice for image.
Select Linux and Ubuntu (64-bit) as shown
Click Continue







On the Screens that follow I recommend the following

  • Memory size: set to at least 4 GB (4096 MB)
  • Hard disk: accept default (Create a virtual disk now)
  • Hard disk file type: accept default (VDI)
  • Storage on physical hard disk: accept default (Dynamically allocated)
  • File Location and Size: 
    • Can accept default name and location, or change it
    • Size of virtual hard drive in megabytes: I recommend 35 GB
You can now double click on the new yellow image name on left panel of VirtualBox window:


This will start your image and then ask where is the disk image is located to start up.
Here you will choose your downloaded Ubuntu iso file in your download folder on your mac from step 1 above. Click on the little folder to right of pulldown menu. Shown here with a red circle around it:

This opens a file open dialog box, find our download named 'ubuntu-17.10-desktop-amd64.iso', open it and click the Start button show above.

Be patient, it will take a while and you may see a Firmware Bug message, just ignore it. It takes a while to get to the next dialog in ubuntu, but it will have two buttons, and you want to press this one:

check box to do updates and click Continue:

Note: if you do not see the Continue button on any of the following steps when expected, make the window for ubuntu a bit wider till you can see it.

On the Installation Type screen, just click Install Now (with the erase disk selection selected)

Click Continue on the 'Write the changes to disks' dialog.

On the Where are you screen, click your timezone if it is not already selected, and click Continue.

On the Keyboard layout screen, click Continue unless you know your keyboard is some other standard.

On the Who are you screen, fill in your name, username, and password (remember your password!)
     I recommend selecting the Log in automatically, since you are running on mac in virtuaBox, I feel I don't need this level of protection since I already have to log into my mac OS. But you can choose what you want. Click Continue when ready.

Now the install process will start. It takes a lot of time (several minutes) so be patient. You will see this dialog when it is done:

click the Restart Now button.
   Be patient.

If you see a message "Please remove the installation medium, then press ENTER". Just press your Enter (Return) key since VirtualBox will automatically remove the CD iso image.

Your Ubuntu Gnome desktop will now show up:



Flickering of ubuntu display need to be solved:

First Issue should now show up, the screen will flash white every once in awhile. 

To fix this, go the the power Icon in the upper right of screen:


Click it and then click your account name, you should now see a Log Off option. Click it.
Then confirm this by clicking Log Out button on the new screen.

After a little while you should be a big button with your name on it. Click it.

Now in the resulting password screen find the little gear just to left of Sign In button and click it.

Choose Ubuntu on Xorg option, then enter your password and click the Sign In button. Wait for it to start back up, and the flickering should be gone, except rarely.

Once you are back to the desktop, I recommend changing two settings in ubuntu.

Disable the lockscreen and the power option that blanks the screen:

Open the settings by going to the icon of 9 white dots in the lower left corner (Show Applications Icon). Type settings into the find box at top and click on the settings tile icon that shows up.

Under the Privacy section, click on the Screen Lock menu, and turn off screen lock as shown here:

Then close the dialog with the little x button to right of Screen Lock dialog box.

Now under the Power selection on left, choose the Blank Screen pulldown and choose Never.

Now close the Settings application with Alt+F4 or use the top right close button with little x.

Setting up VirtualBox Guest additions:

Installing Guest Additions the normal way is another issue with Ubuntu 17.10 in VirtualBox, since the will not install and gets errors. To solve this we have to install the previous editions as follows:

Now you are going to go to your mac and open a browser. We are going to download a older versions of Guest Additions to your download folder. Type the following URL into your browser, or link from this article here:    http://download.virtualbox.org/virtualbox

scroll down and click the version number 5.1.30/

click on VBoxGuestAdditions_5.1.30.iso to download it to your downloads folder on your mac.

Switch back to your ubuntu window, at the bottom you will see a lot of icons. Find the one that looks like a CD-Rom disk, click it and you will see a menu pop up with the top option as 
Choose disk image...

Click on Choose disk image...
In the resulting open dialog, go to your downloads area and select the VBoxGuestAdditions_5.1.30.iso  file and open it.

after a little time, you should see the following dialog pop up:


Click on the Run button, enter your password when prompted and click authenticate.

When you see this:

Then press Return key as instructed.

Note: if later you get a pop up dialog asking to update the Guest Additions to the next version, close it and do not update.

Guest editions is now installed, we have to change some settings in VirtualBox to make use of it.

Click the power icon in the upper right of ubuntu window, then click the power icon in the resulting dialog and click to power off.  

Go back to the VirtualBox window, it should be open the whole time you are running Ubuntu.

You image will still be selected in yellow, go to the top and click the settings gear button.

Under the General sections, go to the advanced tab and turn on Shared Clipboard to the bidirectional setting.

Click OK.

Go back to the settings and click it. Choose the Shared Folder section:


Click the little folder with the plus to the far right. Select the arrow on the folder path text box and select other. Now pick the folder you wish to share with your ubuntu image, I usually share my main drive folder or my user folder.

Select Automount and click OK. (automount will automatically mount the mac folder into the ubuntu image when you start it)

Click OK to leave the settings.

Guest Additions are now all set.

Getting Ubuntu 17 to take advantage of full display resolution:

Before going back to ubuntu, go back to settings for your image in VirtualBox.

Go to the display settings area.

I recommend changing the display memory to 32 MB.

Now turn on the Use unscaled hiDPI setting. This will allow ubuntu to map to every pixel on a high DPI display like the retina mac screens.

Click OK.

Now double click on your image tile to power it up.


You will see that everything in the ubuntu desktop is really small now. That means the hiDPI setting is working.

Now open settings in Ubuntu as before. go to Devices/Displays. If you have a small window the option we are looking for will not be listed. We are looking for the Scale option.

To fix this, we need to make the window bigger. Go to the bottom of the ubuntu window display icon, click on it and choose Virtual Screen 1, go to the largest setting of 1920 x 1200


You should see a setting option now under devices/displays that has a 200% button, click it and then click the Apply button in upper right corner. Click the keep changes button.

NOTE: This 200% scale setting will not keep when you change the window size  So you may have to go back into settings and choose 200% again. Pick a good window size and keep it. I recommend making the size almost fill the screen

Also note that if the fonts are too big for your taste, you can install the app Gnome Tweak and change the font size scale to a number less than 1.0 to make them smaller.

BUG:  when you go to full screen width window with the full screen cmd+f option or using the full screen mac icon, there is a bug and instead of your window being full screen, it will be 1/4 full screen width. We will have to wait for Ubuntu or VirtualBox to fix this in a future release.

Please enjoy Ubuntu now on your mac.   

9/26/17

Installing iPython Notebook for python/data structures class

iPython Notebook is a way of running python from within a 'word processor' like document where you can mix code with formatted text to create a document that can be published for other people to view, or to download and modify.

To install iPython Notebook, open terminal on mac, or cmd.exe or powershell on pc.

PC:  type:  
pip install "ipython[notebook]"
MAC: type:
pip3 install "ipython[notebook]"

iPython Notebook runs in your browser, but you must start up a server on your computer.
To do this open terminal, cmd.exe, or powershell as you did did to install.
Then type
ipython notebook

This will start the server, keep the powershell open, until you are all done with the notebook.
then switch back to the command window and do Ctrl+c on either mac or pc to shudown server,
then you can close the command window program terminal, cmd.exe, or powershell
if closing the window did not close the program.


Website dedicated to iPython Notebook: