Dealing with iterables¶
Without going too deeply into specifics, in python, an iterable is
basically an object that you can have in a for
loop:
for item in my_iterable:
print(item)
Amongst others, lists, tuples, and ranges are iterables, as are strings.
The key is that these objects have a special internal method, an
iterator, attached to them. This iterator is responsible for keeping
track of the index during the iteration, and serving the objects in the
iterable one by one to the for
loop. When writing our own iterable,
we will look under the hood, and see how this all works at the C level.
For now, we are going to discuss only, how we can consume the content
of an iterable in the C code.
Iterating over built-in types¶
In order to demonstrate the use of an iterator, we are going to write a function that sums the square of the values in an iterable. The python version of the function could be something like this:
def sumsq(some_iterable):
return sum([item**2 for item in some_iterable])
sumsq([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
385
In C, the key is in the snippet
mp_obj_iter_buf_t iter_buf;
mp_obj_t item, iterable = mp_getiter(o_in, &iter_buf);
while ((item = mp_iternext(iterable)) != MP_OBJ_STOP_ITERATION) {
// do something with the item just retrieved
}
This is more or less the equivalent of the for item in some_iterable
instruction. In C, the mp_obj_t
object o_in
is the argument of
our python function, which is turned into an iterable by passing it into
the mp_getiter
function. This function also needs a buffer object
that is of type mp_obj_iter_buf_t
. The buffer type is defined in
obj.h
as
typedef struct _mp_obj_iter_buf_t {
mp_obj_base_t base;
mp_obj_t buf[3];
} mp_obj_iter_buf_t;
where .buf[2]
holds the index value, and this is how mp_iternext
keeps track of the position in the loop.
Once item
is retrieved, the rest of the code is trivial: you do
whatever you want to do with the value, and return at the very end.
Now, what happens, if you pass a non-iterable object to the function?
For a while, nothing. Everything will work till the point
item = mp_iternext(iterable)
, where the interpreter will raise a
TypeError
exception. So, on the python console, you can either
enclose your function in a
try:
sumsq(some_iterable)
except TypeError:
print('something went terribly wrong`)
construct, or you can inspect the type of the variable at the C level.
Unfortunately, there does not seem to be a type identifier for iterables
in general, so you have to check, whether the argument is a list, tuple,
range, etc. This can be done by calling the mp_obj_is_type
macro,
and see which Boolean it returns, if you pass &mp_type_tuple
,
&mp_type_list
, &mp_type_range
etc. to it, as we discussed in the
section Object representation.
The complete code listing of consumeiterable.c
follows below. If you
ask me, this is a lot of code just to replace a python one-liner.
https://github.com/v923z/micropython-usermod/tree/master/snippets/consumeiterable/consumeiterable.c
#include "py/obj.h"
#include "py/runtime.h"
STATIC mp_obj_t consumeiterable_sumsq(mp_obj_t o_in) {
mp_float_t _sum = 0.0, itemf;
mp_obj_iter_buf_t iter_buf;
mp_obj_t item, iterable = mp_getiter(o_in, &iter_buf);
while ((item = mp_iternext(iterable)) != MP_OBJ_STOP_ITERATION) {
itemf = mp_obj_get_float(item);
_sum += itemf*itemf;
}
return mp_obj_new_float(_sum);
}
STATIC MP_DEFINE_CONST_FUN_OBJ_1(consumeiterable_sumsq_obj, consumeiterable_sumsq);
STATIC const mp_rom_map_elem_t consumeiterable_module_globals_table[] = {
{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_consumeiterable) },
{ MP_ROM_QSTR(MP_QSTR_sumsq), MP_ROM_PTR(&consumeiterable_sumsq_obj) },
};
STATIC MP_DEFINE_CONST_DICT(consumeiterable_module_globals, consumeiterable_module_globals_table);
const mp_obj_module_t consumeiterable_user_cmodule = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t*)&consumeiterable_module_globals,
};
MP_REGISTER_MODULE(MP_QSTR_consumeiterable, consumeiterable_user_cmodule, MODULE_CONSUMEITERABLE_ENABLED);
https://github.com/v923z/micropython-usermod/tree/master/snippets/consumeiterable/micropython.mk
USERMODULES_DIR := $(USERMOD_DIR)
# Add all C files to SRC_USERMOD.
SRC_USERMOD += $(USERMODULES_DIR)/consumeiterable.c
CFLAGS_USERMOD += -I$(USERMODULES_DIR)
!make clean
!make USER_C_MODULES=../../../usermod/snippets CFLAGS_EXTRA=-DMODULE_CONSUMEITERABLE_ENABLED=1 all
%%micropython
import consumeiterable
a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print(a)
print(consumeiterable.sumsq(a))
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
385.0
Returning iterables¶
Let us suppose that the result of some operation is an iterable, e.g., a tuple, or a list. How would we return such an object? How about a function that returns the powers of its argument? In python
def powerit(base, exponent):
return [base**e for e in range(0, exponent+1)]
powerit(2, 10)
[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]
and in C,
https://github.com/v923z/micropython-usermod/tree/master/snippets/returniterable/returniterable.c
#include "py/obj.h"
#include "py/runtime.h"
STATIC mp_obj_t powers_iterable(mp_obj_t base, mp_obj_t exponent) {
int e = mp_obj_get_int(exponent);
mp_obj_t tuple[e+1];
int b = mp_obj_get_int(base), ba = 1;
for(int i=0; i <= e; i++) {
tuple[i] = mp_obj_new_int(ba);
ba *= b;
}
return mp_obj_new_tuple(e+1, tuple);
}
STATIC MP_DEFINE_CONST_FUN_OBJ_2(powers_iterable_obj, powers_iterable);
STATIC const mp_rom_map_elem_t returniterable_module_globals_table[] = {
{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_returniterable) },
{ MP_ROM_QSTR(MP_QSTR_powers), MP_ROM_PTR(&powers_iterable_obj) },
};
STATIC MP_DEFINE_CONST_DICT(returniterable_module_globals, returniterable_module_globals_table);
const mp_obj_module_t returniterable_user_cmodule = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t*)&returniterable_module_globals,
};
MP_REGISTER_MODULE(MP_QSTR_returniterable, returniterable_user_cmodule, MODULE_RETURNITERABLE_ENABLED);
As everything else, the elements of tuples and lists are objects of type
mp_obj_t
, so, after finding out how far we have got to go with the
exponents, we declare an array of the required length. Values are
generated and assigned in the for
loop. Since on the left hand side
of the assignment we have an mp_obj_t
, we convert the results with
mp_obj_new_int
. Once we are done with the computations, we return
the array with mp_obj_new_tuple
. This functions takes the array as
the second argument, while the first argument specifies the length.
If you happen to want to return a list instead of a tuple, all you have
to do is use mp_obj_new_list
instead at the very end.
https://github.com/v923z/micropython-usermod/tree/master/snippets/returniterable/micropython.mk
USERMODULES_DIR := $(USERMOD_DIR)
# Add all C files to SRC_USERMOD.
SRC_USERMOD += $(USERMODULES_DIR)/returniterable.c
CFLAGS_USERMOD += -I$(USERMODULES_DIR)
!make clean
!make USER_C_MODULES=../../../usermod/snippets CFLAGS_EXTRA=-DMODULE_RETURNITERABLE_ENABLED=1 all
%%micropython
import returniterable
print(returniterable.powers(3, 10))
(1, 3, 9, 27, 81, 243, 729, 2187, 6561, 19683, 59049)
Creating iterables¶
Having seen how we can consume the elements in an iterable, it is time
to explore what this .getiter
magic is doing. So, let us create a
new type, itarray
, and make it iterable! This new type will have a
constructor method,square
, generating 16-bit integers, where the
values are simply the squares of the indices, i.e., 1, 4, 9, 16… We are
interested only in the iterability of the object, and for this reason,
we will implement only the .getiter
special method, and skip
.binary_op
, and .unary_op
. If needed, these can easily be added
based on the discussion in Special methods of classes.
Before listing the complete code, we discuss the relevant code snippets.
The first chunk is the assignment of .getiter
in the
iterable_array_type
structure. .getiter
will be made equal to a
function called iterarray_getiter
, which simply returns
mp_obj_new_itarray_iterator
. Why can’t we simply assign
mp_obj_new_itarray_iterator
, instead of wrapping it in
iterarray_getiter
? The reason for that is that iterarray_getiter
has a strict signature, and we want to pass an extra argument, 0. This
is nothing but the very first index in the sequence.
STATIC mp_obj_t itarray_getiter(mp_obj_t o_in, mp_obj_iter_buf_t *iter_buf) {
return mp_obj_new_itarray_iterator(o_in, 0, iter_buf);
}
const mp_obj_type_t iterable_array_type = {
{ &mp_type_type },
.name = MP_QSTR_itarray,
.print = itarray_print,
.make_new = itarray_make_new,
.getiter = itarray_getiter,
};
So, it appears that we have to scrutinise
mp_obj_new_itarray_iterator
. This is a special object type in
micropython, with a base type of mp_type_polymorph_iter
. In
addition, it holds a pointer to the __next__
method, which is
itarray_iternext
in this case, stores a pointer to the variable (the
one that we are iterating over), and the current index (which we
initialised to 0 in mp_obj_new_itarray_iterator
).
mp_obj_t mp_obj_new_itarray_iterator(mp_obj_t itarray, size_t cur, mp_obj_iter_buf_t *iter_buf) {
assert(sizeof(mp_obj_itarray_it_t) <= sizeof(mp_obj_iter_buf_t));
mp_obj_itarray_it_t *o = (mp_obj_itarray_it_t*)iter_buf;
o->base.type = &mp_type_polymorph_iter;
o->iternext = itarray_iternext;
o->itarray = itarray;
o->cur = cur;
return MP_OBJ_FROM_PTR(o);
}
mp_obj_new_itarray_iterator
is not much more than a declaration and
assignments. The object that we return is of type
mp_obj_itarray_it_t
, which has the above-mentioned structure
// itarray iterator
typedef struct _mp_obj_itarray_it_t {
mp_obj_base_t base;
mp_fun_1_t iternext;
mp_obj_t itarray;
size_t cur;
} mp_obj_itarray_it_t;
mp_obj_t itarray_iternext(mp_obj_t self_in) {
mp_obj_itarray_it_t *self = MP_OBJ_TO_PTR(self_in);
itarray_obj_t *itarray = MP_OBJ_TO_PTR(self->itarray);
if (self->cur < itarray->len) {
// read the current value
uint16_t *arr = itarray->elements;
mp_obj_t o_out = MP_OBJ_NEW_SMALL_INT(arr[self->cur]);
self->cur += 1;
return o_out;
} else {
return MP_OBJ_STOP_ITERATION;
}
}
Now, the complete code in one chunk:
https://github.com/v923z/micropython-usermod/tree/master/snippets/makeiterable/makeiterable.c
#include <stdlib.h>
#include "py/obj.h"
#include "py/runtime.h"
typedef struct _itarray_obj_t {
mp_obj_base_t base;
mp_fun_1_t iternext;
uint16_t *elements;
size_t len;
} itarray_obj_t;
const mp_obj_type_t iterable_array_type;
mp_obj_t mp_obj_new_itarray_iterator(mp_obj_t , size_t , mp_obj_iter_buf_t *);
STATIC void itarray_print(const mp_print_t *print, mp_obj_t self_in, mp_print_kind_t kind) {
(void)kind;
itarray_obj_t *self = MP_OBJ_TO_PTR(self_in);
mp_print_str(print, "itarray: ");
uint16_t i;
for(i=0; i < self->len-1; i++) {
mp_obj_print_helper(print, mp_obj_new_int(self->elements[i]), PRINT_REPR);
mp_print_str(print, ", ");
}
mp_obj_print_helper(print, mp_obj_new_int(self->elements[i]), PRINT_REPR);
}
STATIC mp_obj_t itarray_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args) {
mp_arg_check_num(n_args, n_kw, 1, 1, true);
itarray_obj_t *self = m_new_obj(itarray_obj_t);
self->base.type = &iterable_array_type;
self->len = mp_obj_get_int(args[0]);
uint16_t *arr = malloc(self->len * sizeof(uint16_t));
for(uint16_t i=0; i < self->len; i++) {
arr[i] = i*i;
}
self->elements = arr;
return MP_OBJ_FROM_PTR(self);
}
STATIC mp_obj_t itarray_getiter(mp_obj_t o_in, mp_obj_iter_buf_t *iter_buf) {
return mp_obj_new_itarray_iterator(o_in, 0, iter_buf);
}
const mp_obj_type_t iterable_array_type = {
{ &mp_type_type },
.name = MP_QSTR_itarray,
.print = itarray_print,
.make_new = itarray_make_new,
.getiter = itarray_getiter,
};
STATIC const mp_rom_map_elem_t makeiterable_module_globals_table[] = {
{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_makeiterable) },
{ MP_OBJ_NEW_QSTR(MP_QSTR_square), (mp_obj_t)&iterable_array_type },
};
STATIC MP_DEFINE_CONST_DICT(makeiterable_module_globals, makeiterable_module_globals_table);
const mp_obj_module_t makeiterable_user_cmodule = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t*)&makeiterable_module_globals,
};
MP_REGISTER_MODULE(MP_QSTR_makeiterable, makeiterable_user_cmodule, MODULE_MAKEITERABLE_ENABLED);
// itarray iterator
typedef struct _mp_obj_itarray_it_t {
mp_obj_base_t base;
mp_fun_1_t iternext;
mp_obj_t itarray;
size_t cur;
} mp_obj_itarray_it_t;
mp_obj_t itarray_iternext(mp_obj_t self_in) {
mp_obj_itarray_it_t *self = MP_OBJ_TO_PTR(self_in);
itarray_obj_t *itarray = MP_OBJ_TO_PTR(self->itarray);
if (self->cur < itarray->len) {
// read the current value
uint16_t *arr = itarray->elements;
mp_obj_t o_out = MP_OBJ_NEW_SMALL_INT(arr[self->cur]);
self->cur += 1;
return o_out;
} else {
return MP_OBJ_STOP_ITERATION;
}
}
mp_obj_t mp_obj_new_itarray_iterator(mp_obj_t itarray, size_t cur, mp_obj_iter_buf_t *iter_buf) {
assert(sizeof(mp_obj_itarray_it_t) <= sizeof(mp_obj_iter_buf_t));
mp_obj_itarray_it_t *o = (mp_obj_itarray_it_t*)iter_buf;
o->base.type = &mp_type_polymorph_iter;
o->iternext = itarray_iternext;
o->itarray = itarray;
o->cur = cur;
return MP_OBJ_FROM_PTR(o);
}
https://github.com/v923z/micropython-usermod/tree/master/snippets/makeiterable/micropython.mk
USERMODULES_DIR := $(USERMOD_DIR)
# Add all C files to SRC_USERMOD.
SRC_USERMOD += $(USERMODULES_DIR)/makeiterable.c
CFLAGS_USERMOD += -I$(USERMODULES_DIR)
!make clean
!make USER_C_MODULES=../../../usermod/snippets CFLAGS_EXTRA=-DMODULE_MAKEITERABLE_ENABLED=1 all
%%micropython
import makeiterable
a = makeiterable.square(15)
print(a)
for j, i in enumerate(a):
if j == 1: print('%dst element: %d'%(j, i))
elif j == 2: print('%dnd element: %d'%(j, i))
elif j == 3: print('%drd element: %d'%(j, i))
else:
print('%dth element: %d'%(j, i))
itarray: 0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196
0th element: 0
1st element: 1
2nd element: 4
3rd element: 9
4th element: 16
5th element: 25
6th element: 36
7th element: 49
8th element: 64
9th element: 81
10th element: 100
11th element: 121
12th element: 144
13th element: 169
14th element: 196
Subscripts¶
We now know, how we construct something that can be passed to a for
loop. This is a good start. But iterables have other very useful
properties. For instance, have you ever wondered, what actually happens
in the following snippet?
a = 'micropython'
a[5]
'p'
a
is a string, therefore, an iterable. Where does the interpreter
know from, that it has got to return p
, when asked for a[5]
? Or
have you ever been curious to know, how the interpreter replaces p
by q
, if
a = [c for c in 'micropyton']
a[5] = 'q'
a
['m', 'i', 'c', 'r', 'o', 'q', 'y', 't', 'o', 'n']
is passed to it? If so, then it is your lucky day, because we are going to make our iterable class be able to deal with such requests.
The code snippets above rely on a single special method, the
subscription. In the C code of micropython, this method is called
.subscr
, and it should be assigned to in the class declaration,
i.e., if we take makeiterable.c
as our basis for the following
discussion, then we would have to extend the iterable_array_type
as
const mp_obj_type_t iterable_array_type {
...
.subscr = itarray_subscr
}
where the signature of itarray_subscr
has the form
STATIC mp_obj_t itarray_subscr(mp_obj_t self_in, mp_obj_t index, mp_obj_t value)
If .subscr
is not implemented, but you are daring enough to call
>>> a[5]
all the same, then the interpreter is going to throw a TypeError
:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'itarray' object isn't subscriptable
So, what happens in the method that we assigned in
iterable_array_type
? A possible scenario is given below:
STATIC mp_obj_t subitarray_subscr(mp_obj_t self_in, mp_obj_t index, mp_obj_t value) {
subitarray_obj_t *self = MP_OBJ_TO_PTR(self_in);
size_t idx = mp_obj_get_int(index);
if(self->len <= idx) {
mp_raise_ValueError("index is out of range");
}
if (value == MP_OBJ_SENTINEL) { // simply return the value at index, no assignment
return MP_OBJ_NEW_SMALL_INT(self->elements[idx]);
} else { // value was passed, replace the element at index
self->elements[idx] = mp_obj_get_int(value);
}
return mp_const_none;
}
subitarray_subscr
takes three arguments: the first is the instance
on which the method is called, i.e., self
. The second is the index,
i.e., what stands in []. And finally, the third argument is the value.
This is what we assign to the element at index idx
, or, when we do
not assign anything (i.e., when we load a value from the iterable),
then value
takes on a special value. If we have
>>> a[5]
on the python console, then the interpreter will automatically assign
value = MP_OBJ_SENTINEL
(this is defined in obj.h
), so that,
though we did not explicitly set anything to it, we can still inspect
value
. This is what happens, when we evaluate
value == MP_OBJ_SENTINEL
: if this statement is true, then we query
for a[5]
. Note that we also implemented some very rudimentary error
checking: we raise an IndexError
, whenever the index is out of
range. We do this by calling
mp_raise_msg(&mp_type_IndexError, "index is out of range");
For a thorough discussion on how to raise exceptions see the Section Error handling.
There is one more thing that we should notice: at the very beginning of the function, in the line
size_t idx = mp_obj_get_int(index);
we call mp_obj_get_int
. This means that any python object with an
integer value is a valid argument, i.e., the following instruction would
still work
%%micropython
a = 'micropython'
b = 5
print(a[b])
p
For compiling, here is the complete code:
#include <stdlib.h>
#include "py/obj.h"
#include "py/runtime.h"
typedef struct _subitarray_obj_t {
mp_obj_base_t base;
mp_fun_1_t iternext;
uint16_t *elements;
size_t len;
} subitarray_obj_t;
const mp_obj_type_t subiterable_array_type;
mp_obj_t mp_obj_new_subitarray_iterator(mp_obj_t , size_t , mp_obj_iter_buf_t *);
STATIC void subitarray_print(const mp_print_t *print, mp_obj_t self_in, mp_print_kind_t kind) {
(void)kind;
subitarray_obj_t *self = MP_OBJ_TO_PTR(self_in);
mp_print_str(print, "subitarray: ");
uint16_t i;
for(i=0; i < self->len-1; i++) {
mp_obj_print_helper(print, mp_obj_new_int(self->elements[i]), PRINT_REPR);
mp_print_str(print, ", ");
}
mp_obj_print_helper(print, mp_obj_new_int(self->elements[i]), PRINT_REPR);
}
STATIC mp_obj_t subitarray_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args) {
mp_arg_check_num(n_args, n_kw, 1, 1, true);
subitarray_obj_t *self = m_new_obj(subitarray_obj_t);
self->base.type = &subiterable_array_type;
self->len = mp_obj_get_int(args[0]);
uint16_t *arr = malloc(self->len * sizeof(uint16_t));
for(uint16_t i=0; i < self->len; i++) {
arr[i] = i*i;
}
self->elements = arr;
return MP_OBJ_FROM_PTR(self);
}
STATIC mp_obj_t subitarray_getiter(mp_obj_t o_in, mp_obj_iter_buf_t *iter_buf) {
return mp_obj_new_subitarray_iterator(o_in, 0, iter_buf);
}
STATIC mp_obj_t subitarray_subscr(mp_obj_t self_in, mp_obj_t index, mp_obj_t value) {
subitarray_obj_t *self = MP_OBJ_TO_PTR(self_in);
size_t idx = mp_obj_get_int(index);
if(self->len <= idx) {
mp_raise_msg(&mp_type_IndexError, "index is out of range");
}
if (value == MP_OBJ_SENTINEL) { // simply return the value at index, no assignment
return MP_OBJ_NEW_SMALL_INT(self->elements[idx]);
} else { // value was passed, replace the element at index
self->elements[idx] = mp_obj_get_int(value);
}
return mp_const_none;
}
const mp_obj_type_t subiterable_array_type = {
{ &mp_type_type },
.name = MP_QSTR_subitarray,
.print = subitarray_print,
.make_new = subitarray_make_new,
.getiter = subitarray_getiter,
.subscr = subitarray_subscr,
};
STATIC const mp_rom_map_elem_t subscriptiterable_module_globals_table[] = {
{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_subscriptiterable) },
{ MP_OBJ_NEW_QSTR(MP_QSTR_square), (mp_obj_t)&subiterable_array_type },
};
STATIC MP_DEFINE_CONST_DICT(subscriptiterable_module_globals, subscriptiterable_module_globals_table);
const mp_obj_module_t subscriptiterable_user_cmodule = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t*)&subscriptiterable_module_globals,
};
MP_REGISTER_MODULE(MP_QSTR_subscriptiterable, subscriptiterable_user_cmodule, MODULE_SUBSCRIPTITERABLE_ENABLED);
// itarray iterator
typedef struct _mp_obj_subitarray_it_t {
mp_obj_base_t base;
mp_fun_1_t iternext;
mp_obj_t subitarray;
size_t cur;
} mp_obj_subitarray_it_t;
mp_obj_t subitarray_iternext(mp_obj_t self_in) {
mp_obj_subitarray_it_t *self = MP_OBJ_TO_PTR(self_in);
subitarray_obj_t *subitarray = MP_OBJ_TO_PTR(self->subitarray);
if (self->cur < subitarray->len) {
// read the current value
uint16_t *arr = subitarray->elements;
mp_obj_t o_out = MP_OBJ_NEW_SMALL_INT(arr[self->cur]);
self->cur += 1;
return o_out;
} else {
return MP_OBJ_STOP_ITERATION;
}
}
mp_obj_t mp_obj_new_subitarray_iterator(mp_obj_t subitarray, size_t cur, mp_obj_iter_buf_t *iter_buf) {
assert(sizeof(mp_obj_subitarray_it_t) <= sizeof(mp_obj_iter_buf_t));
mp_obj_subitarray_it_t *o = (mp_obj_subitarray_it_t*)iter_buf;
o->base.type = &mp_type_polymorph_iter;
o->iternext = subitarray_iternext;
o->subitarray = subitarray;
o->cur = cur;
return MP_OBJ_FROM_PTR(o);
}
https://github.com/v923z/micropython-usermod/tree/master/snippets/subscriptiterable/micropython.mk
USERMODULES_DIR := $(USERMOD_DIR)
# Add all C files to SRC_USERMOD.
SRC_USERMOD += $(USERMODULES_DIR)/subscriptiterable.c
CFLAGS_USERMOD += -I$(USERMODULES_DIR)
!make clean
!make USER_C_MODULES=../../../usermod/snippets CFLAGS_EXTRA=-DMODULE_SUBSCRIPTITERABLE_ENABLED=1 all
%%micropython
import subscriptiterable
a = subscriptiterable.square(15)
print(a)
print('the fourth element is %d'%a[3])
a[10] = 0
print(a)
subitarray: 0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196
the fourth element is 9
subitarray: 0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 0, 121, 144, 169, 196
Index reversing¶
Now, the code above works for non-negative indices, but in python it is quite customary to have something like
a = 'micropython'
a[-2]
'o'
which is equivalent to querying for the last but one element (second
from the right) in the iterable. Knowing how long the iterable is (this
is stored in self->len
), it is a trivial matter to modify our code
in such a way that it can return the values at negative indices.
Slicing¶
In the previous two sections we have worked with single elements of an
iterable. But python wouldn’t be python without slices. Slices are index
ranges specified in a start:end:step
format. Taking our earlier
example, we can print every second character in micropython
by
a = 'micropython'
a[0:8:2]
'mcoy'
This behaviour is also part of the .subscr
special method. Let us
implement it, shall we? In order to simplify the discussion, we will
treat one case only: returning values, and we return a new instance of
the array, if a slice was requested, while a single number, if we passed
a single index.
Since we want to return an array if the indices stem from a slice, we
split our original subscriptitarray_make_new
function, and separate
those parts that reserve space for the array from those that do the
assignments.
It shouldn’t come as a surprise that we have to modify the function that
was hooked up to .subscr
. Let us take a look at the following
snippet:
STATIC mp_obj_t sliceitarray_subscr(mp_obj_t self_in, mp_obj_t index, mp_obj_t value) {
sliceitarray_obj_t *self = MP_OBJ_TO_PTR(self_in);
if (value == MP_OBJ_SENTINEL) { // simply return the values at index, no assignment
#if MICROPY_PY_BUILTINS_SLICE
if (mp_obj_is_type(index, &mp_type_slice)) {
mp_bound_slice_t slice;
mp_seq_get_fast_slice_indexes(self->len, index, &slice);
uint16_t len = (slice.stop - slice.start) / slice.step;
sliceitarray_obj_t *res = create_new_sliceitarray(len);
for(size_t i=0; i < len; i++) {
res->elements[i] = self->elements[slice.start+i*slice.step];
}
return MP_OBJ_FROM_PTR(res);
}
#endif
// we have a single index, return a single number
size_t idx = mp_obj_get_int(index);
return MP_OBJ_NEW_SMALL_INT(self->elements[idx]);
} else { // do not deal with assignment, bail out
return mp_const_none;
}
return mp_const_none;
}
As advertised, we treat only the case, when value
is empty, i.e., it
is equal to an MP_OBJ_SENTINEL
. Now, there is no point in trying to
read out the parameters of a slice, if the slice object is not even
defined, is there? This is the case for the minimal ports. So, in order
to prevent nasty things from happening, we insert the #if/#endif
macro with the parameter MICROPY_PY_BUILTINS_SLICE
. Provided that
MICROPY_PY_BUILTINS_SLICE
is defined, we inspect the index, and find
out if it is a slice by calling
mp_obj_is_type(index, &mp_type_slice)
If so, we attempt to load the slice parameters into the slice
object
with
mp_seq_get_fast_slice_indexes(self->len, index, &slice)
The function mp_seq_get_fast_slice_indexes
returns Boolean true
,
if the increment in the slice is 1, and false
otherwise. For the
goal that we are trying to pursue here, it doesn’t matter what the step
size is, so we don’t care about the return value. But the main purpose
of the function is actually something different: the function expands
the start:end:step
slice into a triplet, and it does so, even if one
or two of the slice parameters are missing. So, start::step
,
start::
, :end:step
etc. will also work. In fact, this is why we
have to pass the length of the array: self->len
will be substituted,
if the :end:
parameter is missing.
Equipped with the values of slice.start
, slice.stop
, and
slice.step
, we can determine the length of the new array, and assign
the values in the for
loop.
https://github.com/v923z/micropython-usermod/tree/master/snippets/sliceiterable/sliceiterable.c
#include <stdlib.h>
#include "py/obj.h"
#include "py/runtime.h"
typedef struct _sliceitarray_obj_t {
mp_obj_base_t base;
mp_fun_1_t iternext;
uint16_t *elements;
size_t len;
} sliceitarray_obj_t;
const mp_obj_type_t sliceiterable_array_type;
mp_obj_t mp_obj_new_sliceitarray_iterator(mp_obj_t , size_t , mp_obj_iter_buf_t *);
STATIC void sliceitarray_print(const mp_print_t *print, mp_obj_t self_in, mp_print_kind_t kind) {
(void)kind;
sliceitarray_obj_t *self = MP_OBJ_TO_PTR(self_in);
mp_print_str(print, "sliceitarray: ");
uint16_t i;
for(i=0; i < self->len-1; i++) {
mp_obj_print_helper(print, mp_obj_new_int(self->elements[i]), PRINT_REPR);
mp_print_str(print, ", ");
}
mp_obj_print_helper(print, mp_obj_new_int(self->elements[i]), PRINT_REPR);
}
sliceitarray_obj_t *create_new_sliceitarray(uint16_t len) {
sliceitarray_obj_t *self = m_new_obj(sliceitarray_obj_t);
self->base.type = &sliceiterable_array_type;
self->len = len;
uint16_t *arr = malloc(self->len * sizeof(uint16_t));
self->elements = arr;
return self;
}
STATIC mp_obj_t sliceitarray_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args) {
mp_arg_check_num(n_args, n_kw, 1, 1, true);
sliceitarray_obj_t *self = create_new_sliceitarray(mp_obj_get_int(args[0]));
for(uint16_t i=0; i < self->len; i++) {
self->elements[i] = i*i;
}
return MP_OBJ_FROM_PTR(self);
}
STATIC mp_obj_t sliceitarray_getiter(mp_obj_t o_in, mp_obj_iter_buf_t *iter_buf) {
return mp_obj_new_sliceitarray_iterator(o_in, 0, iter_buf);
}
STATIC mp_obj_t sliceitarray_subscr(mp_obj_t self_in, mp_obj_t index, mp_obj_t value) {
sliceitarray_obj_t *self = MP_OBJ_TO_PTR(self_in);
if (value == MP_OBJ_SENTINEL) { // simply return the values at index, no assignment
#if MICROPY_PY_BUILTINS_SLICE
if (mp_obj_is_type(index, &mp_type_slice)) {
mp_bound_slice_t slice;
mp_seq_get_fast_slice_indexes(self->len, index, &slice);
printf("start: %ld, stop: %ld, step: %ld\n", slice.start, slice.stop, slice.step);
uint16_t len = (slice.stop - slice.start + slice.step - 1) / slice.step;
sliceitarray_obj_t *res = create_new_sliceitarray(len);
for(size_t i=0; i < len; i++) {
res->elements[i] = self->elements[slice.start+i*slice.step];
}
return MP_OBJ_FROM_PTR(res);
}
#endif
// we have a single index, return a single number
size_t idx = mp_obj_get_int(index);
return MP_OBJ_NEW_SMALL_INT(self->elements[idx]);
} else { // do not deal with assignment, bail out
return mp_const_none;
}
return mp_const_none;
}
const mp_obj_type_t sliceiterable_array_type = {
{ &mp_type_type },
.name = MP_QSTR_sliceitarray,
.print = sliceitarray_print,
.make_new = sliceitarray_make_new,
.getiter = sliceitarray_getiter,
.subscr = sliceitarray_subscr,
};
STATIC const mp_rom_map_elem_t sliceiterable_module_globals_table[] = {
{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_sliceiterable) },
{ MP_OBJ_NEW_QSTR(MP_QSTR_square), (mp_obj_t)&sliceiterable_array_type },
};
STATIC MP_DEFINE_CONST_DICT(sliceiterable_module_globals, sliceiterable_module_globals_table);
const mp_obj_module_t sliceiterable_user_cmodule = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t*)&sliceiterable_module_globals,
};
MP_REGISTER_MODULE(MP_QSTR_sliceiterable, sliceiterable_user_cmodule, MODULE_SLICEITERABLE_ENABLED);
// itarray iterator
typedef struct _mp_obj_sliceitarray_it_t {
mp_obj_base_t base;
mp_fun_1_t iternext;
mp_obj_t sliceitarray;
size_t cur;
} mp_obj_sliceitarray_it_t;
mp_obj_t sliceitarray_iternext(mp_obj_t self_in) {
mp_obj_sliceitarray_it_t *self = MP_OBJ_TO_PTR(self_in);
sliceitarray_obj_t *sliceitarray = MP_OBJ_TO_PTR(self->sliceitarray);
if (self->cur < sliceitarray->len) {
// read the current value
uint16_t *arr = sliceitarray->elements;
mp_obj_t o_out = MP_OBJ_NEW_SMALL_INT(arr[self->cur]);
self->cur += 1;
return o_out;
} else {
return MP_OBJ_STOP_ITERATION;
}
}
mp_obj_t mp_obj_new_sliceitarray_iterator(mp_obj_t sliceitarray, size_t cur, mp_obj_iter_buf_t *iter_buf) {
assert(sizeof(mp_obj_sliceitarray_it_t) <= sizeof(mp_obj_iter_buf_t));
mp_obj_sliceitarray_it_t *o = (mp_obj_sliceitarray_it_t*)iter_buf;
o->base.type = &mp_type_polymorph_iter;
o->iternext = sliceitarray_iternext;
o->sliceitarray = sliceitarray;
o->cur = cur;
return MP_OBJ_FROM_PTR(o);
}
https://github.com/v923z/micropython-usermod/tree/master/snippets/sliceiterable/micropython.mk
USERMODULES_DIR := $(USERMOD_DIR)
# Add all C files to SRC_USERMOD.
SRC_USERMOD += $(USERMODULES_DIR)/sliceiterable.c
CFLAGS_USERMOD += -I$(USERMODULES_DIR)
# !make clean
!make USER_C_MODULES=../../../usermod/snippets CFLAGS_EXTRA=-DMODULE_SLICEITERABLE_ENABLED=1 all
%%micropython
import sliceiterable
a = sliceiterable.square(20)
print(a)
print(a[1:15:3])
sliceitarray: 0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361
start: 1, stop: 15, step: 3
sliceitarray: 1, 16, 49, 100, 169
A word of caution is in order here: if the step size is negative, the
array is reversed. This means that slice.start
is larger than
slice.stop
, and when we calculate the length of the new array, we
end up with a negative number. Just saying.