logging in continouous loop

What would be a good way to create logs (with python logging module) inside a constant running loop, without producing a large amount of useless lo开发者_如何学编程g-files? An example would be a loop that constant list a folder, and does some action when it sees a file of a specific type. I want to log that no files were found, or files were found but of a wrong type, without logging that same line constantly for each folder check, as it might run many times a second.

---

Create a Handler that subclasses whatever other functionality you need. Store either the last, or all the previously logged messages that you don't want to emit again:

def make_filetype_aware_handler(handler_class):
    class DontRepeatFiletypeHandler(handler_class):

        def __init__(self, *args, **kwds):
            super().__init__(*args, **kwds)
            self.previous_types = set()

        def emit(self, record):
            if not record.file_type in self.previous_types:
                self.previous_types.add(record.file_type)
                super().emit(record)
    return DontRepeatFiletypeHandler

FiletypeStreamHandler = make_filetype_aware_handler(logging.StreamHandler)

logger = logging.getLogger()
logger.addHandler(FiletypeStreamHandler(sys.stderr))
logger.debug('Found file of type %(file_type)', file_type='x-type/zomg')

---

import logging
logger = logging.getLogger(test)
# logging to a file
hdlr = logging.FileHandler(test.log)
formatter = logging.Formatter('%(asctime)s %(filename)s %(lineno)s %(levelname)s %    (message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr) 
logger.setLevel(logging.DEBUG) 

Then in the loop, you have to check for file type and if file is present or not. Then add :

logger.debug('File Type : %s ' % file_type)

also

if file_present:
    logger.debug('File Present : %s ' % present)

---

Log the less important events with a lower precedence, like DEBUG. See setLevel and SysLogHandler.

At development time set the level to DEBUG, and as your application matures, set it to more reasonable values like INFO or ERROR.

Your app should do something about the errors, like remove files of the wrong type and/or create lacking files; or move the wrongfully configured directories from the job polling to a quarantine location, so your log will not be flooded.

---

My understanding is that you are trying to limit logging the same message over and over again.

If this is your issue, I would create a a set of file_types you have already logged. However you need to be careful, if this is going to run forever, you will eventually crash..

from sets import Set
logged = Set()
while yourCondition:
   file_type = get_next_file_type()
   needToLog = #determine if you need to log this thing

   if needToLog and (not file_type in logged):
      logger.info("BAH! " + file_type)
      logged.add(file_type)

---

A bit of a hack but much easier is "misusing" functools lru_cache.

from functools import lru_cache
from logging import getLogger

# Keep track of 10 different messages and then warn again
@lru_cache(10)
def warn_once(logger: Logger, msg: str):
    logger.warning(msg)

You can increase the 10 to suppress more if required or set it to None to store suppress everything duplicated.