Wednesday, November 19, 2014

pytest fixtures: When does it make sense to use them?

Just a bit of background: pytest (http://pytest.org) is one of the main Python testing frameworks, and it provides many new ways on how to write tests (compared with xUnit based frameworks), so, the idea here is exploring a bit on making use of one of its ideas: fixtures.

Personally, I think fixtures are a pretty good idea, but seeing some real-world code with it has led me to believe it's often abused...

So, below I'll try to list what I think are the PROs and CONs of fixtures and give some examples to back up those points...

PROs: 

  • It's a good way to provide setup/tear down for tests with little boilerplate code.

The example below (which is based on pytest-qt: https://github.com/nicoddemus/pytest-qt) shows a nice example where fixtures are used to setup the QApplication and provide an API to deal with testing Qt.

  
from PyQt4 import QtGui
from PyQt4.QtGui import QPushButton
import pytest

@pytest.yield_fixture(scope='session')
def qapp():
    app = QtGui.QApplication.instance()
    if app is None:
        app = QtGui.QApplication([])
        yield app
        app.exit()
    else:
        yield app

class QtBot(object):

    def click(self, widget):
        widget.click()
        
@pytest.yield_fixture
def qtbot(qapp, request):
    result = QtBot()
    yield result
    qapp.closeAllWindows()

def test_button_clicked(qtbot):
    button = QPushButton()
    clicked = [False]
    def on_clicked():
        clicked[0] = True

    button.clicked.connect(on_clicked)
    qtbot.click(button)
    assert clicked[0]


  • autouse is especially useful for providing global setup/tear down affecting tests without doing any change on existing tests.

The example below shows a fixture which verifies that after each test all files are closed (it's added by default to all tests by using autouse=True to make sure no test has such a leak).

  
import os
import psutil
import pytest

@pytest.fixture(autouse=True)
def check_no_files_open(request):
    process = psutil.Process(os.getpid())
    open_files = set(tup[0] for tup in process.open_files())

    def check():
        assert set(tup[0] for tup in process.open_files()) == open_files

    request.addfinalizer(check)

def test_create_array(tmpdir): # tmpdir is also a nice fixture which creates a temporary dir for us and gives an easy to use API.
    stream = open(os.path.join(str(tmpdir.mkdir("sub")), 'test.txt'), 'w')
    test_create_array.stream = stream  # Example to keep handle open to make test fail


Now, on to the CONs of fixtures...

  • Fixtures can make the code less explicit and harder to follow.

  
--- my_window.py file:

from PyQt4.QtCore import QSize
from PyQt4 import QtGui

class MyWindow(QtGui.QWidget):
    def sizeHint(self, *args, **kwargs):
        return QSize(200, 200)

--- conftest.py file:

import pytest
@pytest.fixture()
def window(qtbot):
    return MyWindow()

--- test_window.py file:

def test_window_size_hint(window):
    size_hint = window.sizeHint()
    assert size_hint.width() == 200


Note that this example uses the qtbot shown in the first example (and that's a good thing), but the bad case is that if we had fixtures coming from many cases, it's hard to know what the window fixture does... in this case, it'd be more straightforward to simply have a test which imports MyWindow and does window = MyWindow() instead of using that fixture... Note that if a custom teardown was needed for the window, it could make sense to create a fixture with a finalizer to do a proper teardown, but in this example, it's clearly too much for too little...

Besides, by just looking at the test, what's this window we're dealing with? Where's it defined? So, if you really want to use fixtures like that, at the very least add some documentation on the type you're expecting to receive in the fixture!

  • It's usually easy to overuse fixtures when a simple function call would do...

The example below shows a Comparator being created where no special setup/teardown is needed and we're just using a stateless object...

  
import pytest

class Comparator(object):

    def assert_almost_equal(self, o1, o2):
        assert abs(o1 - o2) < 0.0001


@pytest.fixture()
def comparator():
    return Comparator()

def test_numbers(comparator):
    comparator.assert_almost_equal(0.00001, 0.00002)

I believe in this case it'd just make much more sense creating a simple 'def assert_almost_equal' function which was imported and used as needed instead of having a fixture to provide this kind of function...

Or, if the Comparator object was indeed needed, the code below would make it clearer what the comparator is, while having it as a parameter in a test makes it much more harder to know what exactly are you getting (mostly because it's pretty hard to reason about parameter types in Python).

  
def test_numbers():
    comparator = Comparator()
    comparator.assert_almost_equal(0.00001, 0.00002)
 
That's it, I think this sums up my current experience in dealing with fixtures -- I think it's a nice mechanism, but has to be used with care because it can be abused and make your code harder to follow!

Now, I'm curious about other points of view too :)

Tuesday, November 11, 2014

Vertical indent guides (PyDev 3.9.0)

The latest PyDev release (3.9.0) is just out. 

The major feature added is that it now has vertical indent guides -- see screenshot below -- they're turned on by default and may be configured in the preferences: PyDev > Editor > Vertical Indent Guide.

This has actually been a long-awaited feature and was added as one of the targets in the last crowdfunding!


Besides this, the 3.9.0 release is packed with many bug-fixes:

  • A critical issue with the minimap when on Ubuntu 12 was fixed. 
  • Some issues in the interactive console (which were introduced due to the latest enhancements related to asynchronous output) were also fixed
  • A bunch of others -- which may be seen at the release notes in http://pydev.org.

Also, this release makes the horizontal scrollbar visible by default again... this is mostly because many users seemed to be confused by not having it (personally, as the editor still scrolls with the cursor and my lines usually aren't that long, it doesn't really bother me, but I can see that many users expect having it -- so, those that want it hidden have to disable it in the minimap preferences and not the other way around).

-- Note that the vertical scroll is still hidden as the minimap is enabled by default.

Thursday, September 25, 2014

Attaching debugger to running process (PyDev 3.8.0)

The latest PyDev 3.8.0 has just been released... along with a bunch of bugfixes, the major feature added is the possibility of attaching the debugger to a running process.

So, I thought about explaining a bit how to use it (and later a bit on how it was done).

The first thing is that the Debug perspective must be activated (as the attach to process menu is only shown by default in the Debug Perspective). Then, when on the debug perspective, select PyDev > Attach to Process (as the image below shows).



When that action is activated, a list with the active processes is shown so that the process we should attach to can be selected (as a note, by default only Python processes are filtered, but if you have an executable running Python under a different name, it's also possible to select it).


After selecting the executable, the user must provide a Python interpreter that's compatible with the interpreter we'll attach to (i.e.: if we're attaching to a 64 bit interpreter, the interpreter selected must also be a 64 bit process -- and likewise for 32 bits).


And if everything went right, we'll be connected with the process after that step (you can note in the screenshot below that in the Debug View we have a process running regularly and we connected to the process through the remote debugger) -- after it's connected, it's possible to pause the running process (through right-click process > suspend)


So, this was how to use it within PyDev... now, I'll explain a bit on the internals as I had a great time implementing that feature :)

The first thing to note is that we currently support (only) Windows and Linux... although the basic idea is the same on both cases (we get a dll loaded in the target process and then execute our attach code through it), the implementations are actually pretty different (as it's Open Source, it can be seen at: https://github.com/fabioz/PyDev.Debugger/tree/development/pydevd_attach_to_process)

So, let me start with the Windows implementation...

In the Windows world, I must thank a bunch of people that came before me in order to make it work:
First, Mario Vilas for Winappdbg: https://github.com/MarioVilas/winappdbg -- mostly, this is a Windows debugger written in Python. We use it to attach a dll to a target process. Then, after the dll is loaded, there's some hand-crafted shellcode created to execute a function from that dll (which is actually different for 32 bits and for 64 bits -- I did spend quite some time here since my assembly knowledge was mostly theoretical, so, it was nice to see it working in practice: that's the GenShellCodeHelper class in add_code_to_python_process.py and it's used in the run_python_code_windows function).

Ok, so, that gives us the basic hackery needed in order to execute some code in a different process (which Winappdbg does through the Windows CreateRemoteThread API) -- so, initially I had a simple version working which did (all in shellcode) an acquire of the Python GIL/run some hand-crafted Python code through PyRun_SimpleString/release GIL... but there are a number of problems with that simple approach: the first one is that although we'll execute some Python code there, we'll do it under a new thread, which under Python 3 meant that this would be no good since we have to initialize the Python threading if it still wasn't initialized. Also, to setup the Debugger we need to call sys.settrace on existing threads (while executing in that thread -- or at least making Python think we're at that thread as there's no API for that)... at that point I decided on having the dll (and not only shellcode).

So, here I must thank the PVTS guys (which actually have an attach to process on Windows which does mixed mode debugging), so, the attach.cpp file which generates our dll is adapted from PVTS to the PyDev use case. It goes through a number of hoops to initialize the threading facility in Python if it still wasn't initialized and does the sys.settrace while having all threads suspended and makes Python think we're actually at the proper thread to make that call... looking at it, I find it really strange that Python itself doesn't have an API for that (it should be easy to do on Python, but it's such a major hack on the debugger because that API is not available).

Now, on to Linux: in Linux the approach is simpler as we reuse a debugger that should be readily available for Linux developers: gdb. Also, because gdb stops threads for us and executes the code in an existing thread, things become much simpler... first, because we're executing in an existing thread, we don't have to start the threading if it's not started -- the only reason this is needed in Windows is because we're executing the code in a new thread in the process, created through CreateRemoteThread -- and also, as gdb has a way to script in Python were we can switch threads and execute something in it while having other threads stopped, we also don't need to do the trick on Python to make it think we're in a thread to execute a sys.settrace as if we were in a thread when in reality we weren't, as we can switch to a thread and really execute code on it.

So, all in all, it should be working properly, although there are a number of caveats... it may fail if we don't have permissions for the CreateRemoteThread on Windows or in Linux it could fail if the ptrace permissions are not set for us to attach with GDB -- and probably a bunch of other things I still didn't think of :)

Still, it's nice to see it working!

Also, the last thanks goes to JetBrains/PyCharm, which helped in sponsoring the work in the debugger -- as I mentioned earlier: http://pydev.blogspot.com.br/2014/08/pydev-370-pydevpycharm-debugger-merge.html the debugger in PyDev/PyCharm is now merged :)

Tuesday, August 26, 2014

PyDev 3.7.0, PyDev/PyCharm Debugger merge, Crowdfunding

PyDev 3.7.0 was just released.

There are some interesting things to talk about in this release...

The first is that the PyDev debugger was merged with the fork which was used in PyCharm. The final code for the debugger (and the interactive console) now lives at: https://github.com/fabioz/PyDev.Debugger. This effort was backed-up by Intellij, and from now on, work on the debugger from either front (PyDev or PyCharm) should benefit both -- pull requests are also very welcome :)

With this merge, PyDev users will gain GEvent debugging and breakpoints at Django templates (but note that the breakpoints can only be added through the LiClipse HTML/Django Templates editor), and in the interactive console front (which was also part of this merge), the asynchronous output and console interrupt are new.

This release also changed the default UI for the PyDev editor (and for LiClipse editors too), so, the minimap (which had a bunch of enhancements) is now turned on by default and the scrollbars are hidden by default -- those that prefer the old behavior must change the settings on the minimap preferences to match the old style.

Also noteworthy is that the code-completion for all letter chars is turned on by default (again, users that want the old behavior have to uncheck that setting from the code completion preferences page), and this release also has a bunch of bugfixes.

Now, I haven't talked about the crowdfunding for keeping the support on PyDev and a new profiler UI (https://sw-brainwy.rhcloud.com/support/pydev-2014) after it finished... well, it didn't reach its full goal -- in practice that means the profiler UI will still be done and users which supported it will receive a license to use it, but it won't be open source... all in all, it wasn't that bad either, it got halfway through its target and many people seemed to like the idea -- in the end I'll know if keeping it on even if not having the full target reached was a good idea or not only after it's commercially available (as the idea is that new licenses will be what will cover for its development expenses and will keep it going afterwards).

A note for profiler contributors is that I still haven't released an early-release version, but I'm working on it :)

As for PyDev, the outcome of the funding also means I have fewer resources to support it than I'd like, but given that LiClipse (http://brainwy.github.io/liclipse/) provides a share of its earnings to support PyDev and latecomers can still contribute through http://pydev.org/, I still hope that I won't need to lower its support (which'd mean taking on other projects in the time I currently have for PyDev), and I think it'll still be possible to do the things outlined in the crowdfunding regarding it.




Thursday, July 10, 2014

New Python profiler UI

So, I wanted to show what I have so far on the profiler front.

Below is its main UI (on a dark palette -- kudos to Qt which makes this pretty easy to do):



The tree shows the attached processes and the CPU tab shows data for the CPU from the process which is being analyzed.

Based on that, it's possible to select a timeframe where stack info is shown (in this case the profiler is analyzing itself). Clicking on a stack opens an internal editor (which can only view the file so far) at the proper place (but the idea is that in the future it'll open the editor in your preferred editor/IDE).

It's possible to start/stop the sampling and the stacks can be grouped either by line or by methods.

Another thing (which is also still incomplete) is provide a view which will show a profile session view graphically... Below is a snapshot of the work on that front so far.



The work is still very much in the beginning, but the sampling view should be usable already -- the plan is to release  the first builds some days after the crowdfunding finishes... thank you to all who are helping to make this possible: https://sw-brainwy.rhcloud.com/support/pydev-2014

Tuesday, June 24, 2014

PyDev 3.6.0 released

PyDev 3.6.0 was just released.

The really major change in this release is that PyDev updated its internal Jython to the latest version, which also allowed PyDev to update its support to the latest pep8.py.

Also, following this, autopep8.py was also integrated (although it seems there's still an issue dealing with non-ascii files -- see: https://sw-brainwy.rhcloud.com/tracker/PyDev/402 for a workaround). Note that unlike pep8.py, autopep8.py doesn't really use the internal Jython (it runs based on the default configured Python interpreter). This is mostly because Jython was too slow when running it (annoyingly so in this use-case), so, there's only the option of running autopep8.py externally.

Also, some 'nice to have' features were added:

Shift+F9 will run the current editor directly in debug mode -- so, no need to Alt+R+G, 1-4 for that anymore :)

Also, when selecting a unit-test to run (Ctrl+F9), if Shift is pressed when the test(s) to run is selected, it'll also run in debug mode (previously it was almost mandatory to create a regular run selecting the tests, killing it and pressing F11 just to re-run the last test in debug mode).

Besides this, there were some bug-fixes (and for those using LiClipse: http://brainwy.github.io/liclipse, it's already updated with the latest PyDev version -- and now it supports launching for other languages and Julia support was added).

Enjoy!

p.s.:  Thank you for all that are supporting the PyDev crowdfunding: https://sw-brainwy.rhcloud.com/support/pydev-2014/

Tuesday, May 20, 2014

PyDev 3.5.0 released

PyDev 3.5.0 is now available...

The main feature in this release relates to improvements to work with py.test (note: those that want to use it have to enable it in the PyUnit preferences -- besides having to install it properly in the used Python installation).

Mainly, the integration was changed so that the py.test xdist plugin can be properly used and module/session scoped fixtures should also work properly. The PyUnit UI in the preferences was also changed to show the preferences related to py.test when it's shown.

Still, there are other improvements: the Model.objects from django 1.7 is now available on code-completion again (as the static analyzer couldn't cope with it there are workarounds in the code), private attributes (those starting with '__') are now shown when hovering over it on the debugger and there's a bunch of other bug-fixes and minor features available (see http://pydev.org for details).

Also, for those that don't know, there's an ongoing crowdfunding to keep PyDev support going on: http://tiny.cc/pydev-2014...

Thursday, May 15, 2014

Crowdfunding for PyDev and new Python profiler UI (for any IDE)

Ok, I just started a crowdfunding project (at http://tiny.cc/pydev-2014) for the continued development of PyDev and the creation of a new profiler UI.

This is actually the 2nd crowdfunding for PyDev after Appcelerator stopped supporting it. Last year a similar crowdfunding allowed me to keep on developing PyDev until now, as well as creating a new PyDev/Eclipse bundle called LiClipse, which added lightweight support for several languages as well a dark theme inside Eclipse.


So, now I'm reaching out again for the PyDev community to allow me to keep on developing PyDev and shaping the Python landscape.


This year when planning what to do and filtering on existing requests, one of the things that stood up was providing proper support for profiling in PyDev.


After tinkering with the idea a bit and trying to find out how I think would be the best way for this workflow, I've settled on creating a separate tool and then integrating it properly in PyDev... not only this approach will allow Pythonistas that don't use PyDev to have a nice profiling companion, but I think it'll also yield better results than doing it all inside of Eclipse -- besides, I'll be able to do it all in Python!


So, this is the main feature I've planned for this year, but there are others too which are more focused on PyDev itself, such as having preferences per-project, migrating to a new workspace, validating docstrings for its type on a test run, external pep8 linter, vertical lines for indentation, attach debugger to running process -- besides the usual bugfixing and support.


So, please help in funding (and sharing) at http://tiny.cc/pydev-2014 to help in making the new profiler UI a reality and to keep PyDev going strong!



Friday, March 28, 2014

Mastering writing code on PyDev

One of these days I was pair coding with a friend -- which was also using PyDev -- and upon looking him code I realized that there are some simple things which everyone should know when writing code on PyDev:

1. Want to write a new line? Use Shift+Enter: This is what I almost always use in PyDev instead of the plain enter to enter a new line.

How it works: It'll emulate moving to the end of the line and pressing enter there.

Tip: It can be used when there's a completion to avoid applying it while going to the next line (as Enter is also used to apply a completion if you don't use Shift+Enter you have to press ESC before to avoid applying it if you use a plain Enter).

2. Want to copy or move some lines? Ctrl+Alt+Up will copy the lines up and Ctrl+Alt+Down will copy them down and then Alt+Up or Alt+Down can be used to move them up or down (so, it's a line-oriented copy/paste which is really handy).

3. Is there a string or comment which is too big and you want to wrap it? Use Ctrl+2, W and it'll wrap it for you (using the number of columns assigned in the print margin preferences).

4. Want to create a docstring for the parameters of a method? Ctrl+1 -> Make Docstring (in a line with the method declaration).

5. Assign parameters to variables? Ctrl+1 -> Assign parameters to attributes (in a line with the method declaration in a class). Alternatively use Ctrl+2, A.

6. Want to select the current word? Shift+Alt+Up will select the current word for you (and if you keep using Shift+Alt+Up it'll then select the enclosing context and Shift+Alt+Down will deselect it again).

7. Rectangular selection: Shift+Alt+A.

8. Want to rename a local token? To do that use Ctrl+2, R (note that for renaming a token in multiple modules you'd use Shift+Alt+R or a package/module in the PyDev Package Explorer would be renamed with F2).

9. To rename something that's not a token (such as a comma) -- if you're in LiClipse (which adds this feature) -- you can do Ctrl+K (or Ctrl+Shift+K to go backwards) multiple times to mark the places you want to rename and then go on to rename it (as a note, Ctrl+Alt+K will unlink one of the occurrences). This can also be used for a substring -- or any character combination for that matter.

10. Do you have some code with multiple tokens that are not found and need to be imported? Ctrl+Shift+O can be used to resolve the multiple missing tokens. Alternatively you could also use Ctrl+1 on a line with a missing import to get the suggestion to add it (if it wasn't already added in a code-completion which added it for you automatically).

11. Like TDD coding? i.e.: write the test first and do the code later. If so, you can usually write the test and use Ctrl+1 to provide a suggestion to create the missing method, module or attribute.

12. PyDev doesn't automatically re-analyze dependent modules when one module changes, so, I've seen people just add some space and save the file again just to force a new code analysis... while this works, there's a specific keybinding: Ctrl+2, C which will force a code-analysis in the current module.

13. When there are many contiguous lines commented and you want to uncomment them, do you see yourself selecting all those lines to uncomment it? Did you know that Ctrl+5 does that promptly for you from any line of that block?

14. Just edited something but don't remember where it was? Ctrl+Q can be used to go to the last place you edited some code.

15. Do you have some code you just copied from somewhere and want to create a module with it? Go to the PyDev package explorer and just paste the code there to create a new module with those contents.

16. Do you have some method call or variable that you want to add to another local variable or field? Use Ctrl+1 (on a line that still doesn't have an assign) and choose assign to local or assign to field.

Besides those tips, print the bindings from: http://pydev.org/manual_adv_keybindings.html and leave it by your side until you don't master them :)

Thursday, March 20, 2014

PyDev 3.4.1: improved interactive console, multiprocess debugging

One of the major features added is that now F2 can be used to send the current line to the interactive console (or the current multi-line selection).

It's close to what was previously used as Ctrl+Alt+Enter, but with some differences:

1. It'll fix the indentation of the code when sending things to the console (so, you shouldn't have syntax errors because of the indentation when using Ctrl+Alt+Enter).

2. When a line is sent, the cursor goes to the next line (this may seem minor, but it's a huge time-saver and makes things much smoother).

3. It's a single keypress! (and that was really hard to decide as all keys seem to be taken -- even F2 -- but I think that F2 as it was before wasn't very useful).

All in all, if you're into doing things interactively, things are nicer with F2.

Note: Ctrl+Alt+Enter can still be used to do an execfile, but aside from that, F2 is the preferred way to send contents to the console.

Credit goes to Ed Catmur and James Blackburn for this feature -- I just integrated it :)

Another major feature is that the debugger now works properly when debugging multiple processes (and when a launch is terminated it'll also kill all subprocesses). This means the previous patch the debugger did to Django is no longer needed... Just F11 to relaunch your last launch and debug multiple processes!

Major feature #3: Ctrl+F9 (which opens a dialog for selecting which tests to run) now works properly with py.test too -- even if tests are not under class.

Major feature #4: pxd and pxi Cython files are now properly handled.

And as the last thing, which isn't an actual feature, but something noteworthy: PyDev no longer changes the default encoding (i.e.: sys.setdefaultencoding). Back when PyDev started to do that (in Python 2.4 I believe) there wasn't much choice to see unicode contents properly in the console, so, PyDev changed the default encoding to be the encoding of the console. Fast forward a bit and Python now provides a way to set the sys.stdout/stderr encodings through PYTHONIOENCODING (Python 2.6 onwards), so, PyDev now only sets that variable and no longer changes the default encoding (the main issue there is that when some application was later deployed, it could have a different default encoding and things could break because of UnicodeDecodeErrors).

Note that this is not all, other things were done too... See: http://pydev.org/ for details -- and note that LiClipse: http://brainwy.github.io/liclipse/ is already updated to the latest PyDev too.

Sunday, March 02, 2014

Should the Python garbage collector be disabled?

Ok, though question... so, first a little bit of background:

The Python garbage collector is useful for collecting reference cycles, but objects are collected by default when their reference count reaches 0, so, most of the time objects will be collected properly and the collector is only useful when you have a cycle.

Also, there's no guarantee when it'll bump in to do a collection, so, if you're doing UI programming (i.e.: using something as Qt), and you use multiple threads, if you have a cycle, it's possible that the cycle is broken on a collect out of the main thread, which can cause your application to crash if an UI object is collected!

In this case, even if you're careful about collecting things, there's always the case where you have an exception and the object goes to sys.exc_info and becomes alive for much more time than you'd intend, so, if you are using an UI framework, at least making sure that you only collect in the UI thread is a must (see below code which helps doing that).

So, personally, I think that in Python the garbage collector should always be turned off (which can even make your code a lot faster in many situations) and the gc module should be used as a debug tool to find cycles which may occur -- and those should be treated as application errors!

weakref.ref() is one of the most useful things for breaking the cycles and if you need references to methods use a WeakMethodRef: http://code.activestate.com/recipes/81253/

Below is some code to make manual garbage-collection (credit to Erik Janssens) -- while developing the method check() should usually return self.debug_cycles() -- if you want you can use the remaining code to leave as a tool to break cycles in a real application if you want to play safe (although I think disabling it altogether is better if you make sure you don't have cycles) ...

Also, while we're talking about cycles and garbage collection, make sure you never override __del__... Python has an optional callable in weakref.ref() which can be used to do things when an object is collected -- and which doesn't have the problems related to __del__.


class GarbageCollector(QObject):
    '''
    Disable automatic garbage collection and instead collect manually
    every INTERVAL milliseconds.

    This is done to ensure that garbage collection only happens in the GUI
    thread, as otherwise Qt can crash.
    '''

    INTERVAL = 10000

    def __init__(self, parent, debug=False):
        QObject.__init__(self, parent)
        self.debug = debug

        self.timer = QTimer(self)
        self.timer.timeout.connect(self.check)

        self.threshold = gc.get_threshold()
        gc.disable()
        self.timer.start(self.INTERVAL)

    def check(self):
        #return self.debug_cycles() # uncomment to just debug cycles
        l0, l1, l2 = gc.get_count()
        if self.debug:
            print ('gc_check called:', l0, l1, l2)
        if l0 > self.threshold[0]:
            num = gc.collect(0)
            if self.debug:
                print ('collecting gen 0, found:', num, 'unreachable')
            if l1 > self.threshold[1]:
                num = gc.collect(1)
                if self.debug:
                    print ('collecting gen 1, found:', num, 'unreachable')
                if l2 > self.threshold[2]:
                    num = gc.collect(2)
                    if self.debug:
                        print ('collecting gen 2, found:', num, 'unreachable')

    def debug_cycles(self):
        gc.set_debug(gc.DEBUG_SAVEALL)
        gc.collect()
        for obj in gc.garbage:
            print (obj, repr(obj), type(obj))

Thursday, February 13, 2014

Configuring Kivy on PyDev

Kivy installation on PyDev is a bit more involved because it requires some environment variables to be set in order to work properly, so, I thought I'd give a step-by-step on where exactly to configure that on PyDev (especially now that LiClipse: http://brainwy.github.io/liclipse/ adds support to the Kivy Language).

Note: The details below use paths based on windows, but it should be similar in other platforms.

1. Download/extract Kivy (for this example D:\bin\Kivy-1.7.2-w32\)

2. Add interpreter in Window > Preferences > Pydev > Interpreters > Python Interpreter (point to: D:\bin\Kivy-1.7.2-w32\Python\python.exe).

Note: For PyDev 3.3.3 onwards, the easier way to go there is doing: Ctrl+3 and writing 'Python interpreter' to open that preferences page (and the same thing can be used to go to a view or even activate some action).

3. Add the Kivy directory to the PYTHONPATH for this interpreter (in the same Python Interpreter page > libraries > add folder > D:\bin\Kivy-1.7.2-w32\kivy)

4. Add 'kivy' to the 'forced builtins' (again in that same page > forced builtins).

5. Add the needed environment variables (in that same page > environment):

GST_REGISTRY = D:\bin\Kivy-1.7.2-w32\gstreamer\registry.bin

GST_PLUGIN_PATH = D:\bin\Kivy-1.7.2-w32\gstreamer\lib\gstreamer-0.10

PATH = D:\bin\Kivy-1.7.2-w32;D:\bin\Kivy-1.7.2-w32\Python;D:\bin\Kivy-1.7.2-w32\gstreamer\bin;D:\bin\Kivy-1.7.2-w32\MinGW\bin;%PATH%

Alternatively, instead of adding those manually to the environment, open a cmd.exe, execute D:\bin\Kivy-1.7.2-w32\kivyenv.bat and then start Eclipse (but then you have to remember to do that manually every time -- or add it to the system environment variables -- note that you have to remember to update it if you move it or upgrade kivy).

After that, it should be possible to go to the pydev package explorer, expand the interpreter node in the tree > system libs > examples, open some example main.py and open it, then, with the editor opened used F9 to run the example (you may have to select which project should be used to get the information on the PYTHONPATH to be used as it's running as an external file).

Tuesday, February 04, 2014

Changing the locals of a frame (frame.f_locals) and persisting results (with ctypes)

Up until now I didn't know of a proper way to change the locals of a frame (out of the normal execution flow in Python), so, when in the PyDev debugger changing a variable wouldn't always work.

So, for instance, if you have a frame (which you could get from a traceback, sys._getframe().f_back, etc), you could get its locals with frame.f_locals, but changing the frame.f_locals (which gives you a dictionary) wouldn't apply the results back to the frame.

This is mostly due to how CPython works: frame.f_locals actually creates a dictionary using PyFrame_FastToLocals, but changes to the dictionary aren't applied back.

Some years ago I had found a way to make it work (see: http://bugs.python.org/issue1654367) through a CPython function: PyFrame_FastToLocals, but up until recently, I thought it needed a modified version of CPython in order to work, now, recently I discovered ctypes can access a lot from the python api (through ctypes.pythonapi):

So, after changing frame.f_locals, it's possible to use ctypes to call PyFrame_LocalsToFast doing:


import ctypes

ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(frame), ctypes.c_int(0))

 

A note: the second parameter (which may be 0 or 1) defines whether we want to erase variables removed from the dict (which would require 1) or not.

So,  the PyDev debugger now incorporates this utility so that if you're running in CPython, it will properly change the variable in a scope when you change a variable :)

Note that this isn't compatible with other Python implementations (this is not something the language dictates how it should work -- probably the ideal would be making frame.f_locals writable).


Tuesday, January 28, 2014

New PyDev release: improved indexing, Kivy support on LiClipse, etc.

PyDev 3.3.3 is now released.

For this release, there are lots of enhancements in many areas (full details on http://pydev.org).

My favorite one is that PyDev will now index completions from compiled modules (i.e.: .pyd files or entries in the 'forced builtins' -- http://pydev.org/manual_101_interpreter.html has more details on what are forced builtins). This was a long due request but it required the many other incremental changes from recent releases to be feasible.

With that, the context-insensitive code completion (the one that'll automatically add an import for the token) will work for libraries such as PyQt, itertools, etc.

Another which is really nice is that in the debugger, changing a local variable works properly (until now, this did work sometimes, but usually it didn't, as it had shortcomings depending on what variable was changed and how it was put in the scope and whether it was the top frame). Now this works even when assigning a local in the Debug Console!

Also in the debugger, now it's possible to mark some functions that you want to ignore in the debugger with a comment: #@DontTrace and when stepping in the debugger will ignore those: this is a huge time saver when debugging to ignore paths which are just scaffolding (and many times get in the way during a debug session).

As for LiClipse, the Kivy language is now supported -- it has syntax highlighting, code-completion, outline and many other goodies you'd expect. Besides this, in the latest release (0.9.7) LiClipse users can benefit from mark occurrences in the created editors and the theming now applies to EGit views too (more details on http://brainwy.github.io/liclipse).

Now, this is just a brief and incomplete summary of the changes. Personally, I think that PyDev improved on so many things in this release that it's a must have update if you're a PyDev/LiClipse user (especially performance-wise) -- I almost thought about making it a version 4.0 coming from 3.2, but I just couldn't skip doing a 3.3.3 version :)

Wednesday, January 08, 2014

Profiling a method on Python

This week I needed to do a profile session and usually on Python I just use the profile module and dump the output stats (of cProfile) in textual mode (which is usually enough to find out about the problem).

Now, this week I had to do some profiling which demanded a bit more, so, researching a bit, it seems that graphviz (http://www.graphviz.org/) can be used to plot the results of the profile session output with the help of gprof2dot (http://gprof2dot.jrfonseca.googlecode.com/git/gprof2dot.py).

So, using the code below (gist from https://gist.github.com/fabioz/8314370), it's possible to profile a function and have a graphical (.svg) output with the results of the profile (besides the usual text output, which I usually save temporarily during a profile session to compare the results from subsequent optimizations).

Hopefully the docstring explains how to use it properly (as well as its dependencies):




Note that it relies on having a .svg viewer installed (I had Inkscape: http://www.inkscape.org/ installed, so, I'm just using it, but there may be better .svg viewers around).

Happy profiling!